Make Your Website Smart: Add AI Search with NLWeb in 10 Minutes — No Coding Needed!

Have you tried to create your own rag pipeline? It can be a lot of work. Imagine you have a website with some content you want to create a rag pipeline based on. You have to get the content on the website rendered. Then you have to extract the text from the web pages that have been rendered. You have to chunk that text. You have to create embeddings. You have to then put them in a vector database. And then you have to start uh figuring out a way to create a query system such that you can retrieve that context from the vector database. Feed it to an LLM to generate a response and then you have a result. And don't forget, you also want a way to make it easy for AI agents to access the content on your website. So that means you also need to set up an MCP server. And that is a lot. It's a lot of steps, a lot of things to set up. What if you have a framework that provides the core tools you need to create a conversational interface for any website? And better yet, what if you could deploy it in one click without having to write a single line of code? In this video, I'm going to show you how to create a rag pipeline for any website for free using AI search powered by N web in just under 10 minutes. and then you can have AI agents and humans interface with it. Let me show you a quick demo of what you achieve at the end of this video. So, I have my personal website here. It's on confidence.sh SH and then I also have a blog on it which has uh a few articles I've written in the past that I want to be accessible to AI agents and also to humans through a chat interface. For this I have an NL web deployment and I'm just going to paste the link in here and uh what you're greeted with is a familiar chat interface. Now, this is different from other chat interfaces in the sense that all of the content on this uh platform is powered by stuff I have on my blog. So, I can come and say what is this blog about? And it's going to say it's a blog by confidence. And you can see some links with references to where he was able to pull this content from the actual articles on my blog. But we can do more than just ask general questions. We can ask for content specific to what's in an article. So I can ask something like how do I fix DNS issues in Docker. All right. And you can see it's able to rewrite the query in such a way that it pulls the relevant content from my blog. And here is an explanation on how to do it. And it also links to the actual article that talks about how to do it. So if I want more uh context, I'm able to go to the article to read it. The best part of this web setup is that it also comes with an MCP server, which makes it easy for me to give AI agents access to content on my website. So I'm just going to open up an MCP client. This could be your uh cloud desktop or chat GPT. I have one that works in the browser which is on playground.ai.cloudfare.com and this is all built using open source models. So I can go select a model. So let's select llama 3.3 and I am going to put in the link of my NL web deployment which is um everything here in the NL web chart. I'm just going to copy this again with a for/mcp and let's connect this to the AI model and we have an ask tool which provides context based on my website to my chat agent here. So, let's say um I'm just going to ask how do I fix course errors and as you can see the agent is calling the ask tool which uh sends the query over and returns a response and that response is used by the LLM to generate an answer which you can see this is how you go about fixing it. So, this is really cool and this is all what I'll be showing you how to set up uh completely free of charge for any website you own using AI search powered by NL web. My name is Confidence and I'm a developer advocate at Cloudflare. Without any delay, let's get started. At this point, you've heard me say NL Web um a couple of times. What is NL Web? Here is a one minute crash course. N web or natural language web is an open-source framework developed by Microsoft that lets websites add a chat style interface powered by AI just like you saw to any website such that users can just ask questions in plain language and get answers based on the content of your own site. And it's important because this helps site owners to stay in control of uh your data while making your web pages um accessible for humans and AI agents making it easy to understand and to interact with. Now you may be wondering why is this important? uh it's important because it helps you as a content owner to be independent of answer engines like uh Google AI search overview or charge GPT and for you to have your own native content discovery system within your website. It also makes it easy for you to save your content to AI crawlers or AI agents in a fair and monetizable way. So, what I'll do quickly is show you how to set this up for your website. And uh the only thing you need really is a free Cloudflare account, which you can sign up for on dash.cloudflare.com. Um you may need to put in a credit card on file. That's because AI search is powered by R2 and to prevent abuse, we do need to have your credit card, but it's not going to be charged. Then, of course, you need a website which is running through Cloudflare. I'll leave resources on how to set up your domain or cloudware um in the description below. So, let's head over to my dash and I'm going to show you how to set all of these up. So, this is going to be on dash.cloudflare.com. If you don't have an account, uh feel free to go and uh sign up for one. It's completely free. And you want to have your domain on Cloudflare. So either you have your site um running on cloudare or you have your domain on cloudare which means your site could sit on any other server or could be hosted somewhere else. I already have my site which is uh confidence.sh added to cloudflare as you can see I have my confidence.sh domain over here and what I want to do now is go to uh compute an AI and then go to AI search. I just wanted to quickly add before moving on that the reason why your site needs to sit behind Cloudflare is so Cloudflare can verify you have ownership of the content of that site in order to create an ML web pipeline for you automatically. So I'm just going to head here to AI search which used to be called Autorag and let's go ahead to create a new pipeline. So I'm going to click on the create button. you want to use the inner web for websites and we have a template that uh automatically does the whole process for you essentially. So here's where you go select um what domain your blog or your site content is. Um I have a couple of domains. This is going to be on confidence.sh and it's not on a subdomain. It's on the root domain itself. So, I'm going to go ahead to click on start indexing. And this is going to spin up the entire pipeline to um index the content on my website and to put it in a vector database uh to create a search pipeline and also to create and deploy an MCP server all using NL web because we selected the NL web template as you saw earlier. So, this is going to take a minute or two depending on how much content you have on your website because all of it is going to be uh extracted and put into our AI search pipeline. So, it's done four pages so far. And let's come back to it when it's done indexing all of the content on my website. All right. And it's done indexing all of the pages on my website, which is cool. and it's created an ML web deployment where I have a chat interface and I can talk to my website using a chat interface. So, I'll go copy this over and let's open that up in a new tab. And of course, I can go ahead to ask questions on uh this website. Uh I can say what what is this site about? And we have stuff there. Um, I can ask does it have any content on running local LLMs and it's going to write run a query that um should fetch relevant context from my website on running localms and you have the response there. So this is just one half of it. The really cool thing about it is that if I wanted to, I could integrate the entire chat interface um in my website and that makes it such that it's a unified experience and I don't have to go open up a new link to be able to chat to the content on my website. So there is a drop down here that says add NL web to your website and this is the code you need to um yeah the snippet you need to copy paste into your website to have this embedded and we can go preview the widget. So let's click on the preview widget you have a search widget which is like a regular search bar but then this has context of my website. So let's say how do how do I fix DNx issues? And you can see it's again pulling information from the article on fixing uh DNS issues in Docker. This is a search widget I could embed in my website and that makes the experience so much better. I can click on the links to open them up as you'd expect. And this is really cool. So this is all done now. Um I can click on the continue to dashboard button and here is the here is the information on the pipeline that has been created so far. I can go see the metrics of search queries that has been performed over time. I also have a playground to test the performance of the current pipeline and of course switch models that are used for generating responses or uh creating the embeddings and uh tweak some of those parameters as well. Awesome. This is how easy it is to integrate um NL web into any website you own using AI search. And uh the cool thing about this is that in the coming weeks we'll make this experience so much better [music] such that um you can automatically integrate this to any website you own that's on Cloudflare by just clicking a button. We're also making the integration to your website better by making the uh widgets more customizable through custom CSS uh having an option for a floating widget and more. And I think one other cool feature I'm looking forward to is the deep research mode through which the LLM is going to be able to perform a deep research based on the content on [music] your website. So that's also going to be coming in the next few weeks. Like I mentioned, I'm going to have links to resources of this video in the description below. So you can go uh check them out. And I would love to see what website you are going to try in a web on. Please let me know in the comments below and I'll see you in the next one. Don't forget to get subscribed. I'll catch you next time. Take care. Bye.