Welcome to openkapow Sign in | Join
in Search

Tutorials

Enjoy our tutorials on building your own robots and mashups.

Creating a basic REST robot that searches Google

In this tutorial we will see how to create, test, publish and use a very simple REST robot using openkapow. The robot will create a REST service that searches Google and returns the first result of the search. This tutorial describes the process in great detail and most of it applies to RSS robots as well, and in some degree also to Clipping robots.

It is assumed that you already have downloaded the robot development environment RoboMaker and registered as a user on openkapow. All images in the tutorial can be viewed at full size, just click on the image you are interested in and it will be opened in a new window. The robot built in this tutorial is downloadable here.

Part 1 - Create a REST robot

Start with opening up RoboMaker, if you are using Windows you will find RoboMaker in your start menu under Kapow RoboSuite Openkapow Edition. In the startup wizard choose the option "Create a REST web service..." and click OK.

In the New Robot Wizard that opens you need to enter the URL the REST robot should start from, in this case we are going to search Google, so we enter "www.google.com" and click Next.

In the next screen of the New Robot Wizard we enter all the input values we want our robot to have. Since we want to search Google we need to know what to search for. This will be an input value to the robot. So write "searchText" in the input value text field and click "Add". This robot only need one input value, so when we have added "searchText", click Next.

On the next screen we add the default value for the "searchText" input value. This is the value we will use while developing the robot. In this case we will need a text to search for in order to get any results from Google. So let us use the term "kapow" during development. Once this robot is done and published to openkapow.com the input value "searchText" will not in any way be hardcoded to "kapow", instead this will be a parameter that anybody using the robot can specify in the url that calls the robot.

We have now defined the input to our REST robot, so it is time to define some output values. The robot should return the first search result from Google. For this first search result we want both the title and the URL, so add "title" and "url" as output values. Then click Finish to generate the new REST robot.

We have now created a very basic REST robot that contains two steps - one "Load Page" and one "Return Response" step - as you can see at the top of RoboMaker. The Load Page step, does just what the name implies, loads the a page, just like if you used a browser to do the same thing. The Return Response step returns data from the robot and this is the data that the REST service will return.

The robot will start out with the last step active, which is indicated by the green highlighting of the step. Click on the "Load Page" step to make it active. In the right of the screen are the settings for the current step, which now should be Load Page. As you can see the URL "www.google.com" is to be loaded in this step.

If you click on the "Return Item" step, all steps before that will be executed, in this case only the "Load Page" step. This means that www.google.com will be loaded into RoboMakers internal browser view. RoboMaker executes the steps during development in the same way as when a robot is run on openkapow.com. This feature is very usefull as you will always see what state the robot is on a particular step. In the right of the screen we can now see the the step configuration for the active step. In that configuration we see that the Return Response step will return an object called RESTOutput.

Let us take a look on the Input Objects in the lower right of the RoboMaker screen. Here we can see how the input and the output we defined in the New Robot Wizard are represented in the robot. The RESTInput object has 10 name-value pairs. In this case we only specified one input value, so the attribute "name1" is set to "searchText" and the attribute "value1" is set to "robot". This means that when somebody calls the robot via an openkapow.com URL using the parameter "searchText" (for example http://www.openkapow.com/....?searchText=kapow") then the value of that parameter will be available within the robot in the "value1" attribute. Note that if the robot has several input parameters the order does not matter, the parameter "searchText" will always be mapped to name1-value1 (in this case). Output Objects work the same way as input objects, and every REST robot uses the output object RESTOutput, which has 10 name-value pairs, just as RESTInput has. For this robot we see that RESTOutput.name1 is set to "title" and RESTOutput.url is set to "url", which are the output values we defined in the New Robot Wizard. The use of both RESTInput and RESTOutput will clarify as we continue this tutorial.

Part 2 - Search Google

We have now created a REST robot, so it is time to add some functionality to it. The robot should search Google and return the first result. So the robot needs to write the content of the input value "searchText" into the Google text field, then the robot should click the "Search" button on Google and finally the robot should read the first search result and return that data. First thing first, we need to make the robot enter the value of the input value "searchText" into the Google text field. To do this click on the Google text field in the browser view.

Notice that the title is highlighted in 4 different places, in the browser view, in the HTML source code, in the HTML path and in the DOM (don't worry if you do not know what all this means, you do not need to know this if you are not going to make much more complicated robots than this one). We have clicked on the "input" HTML tag which represents the search text input field. To make the robot enter the searchText into this text field simply right click on the text field in the browser view and then take "Enter Text from Attribute", choose the object "RESTInput" and the attribute "RESTInput.value1 (searchText)".

Several things have now happened. A new step called "Enter Value 1" have been inserted between Load Page and Return Response. If you look in the browser view you will see that the text field now contains the value "kapow", which we defined as the searchText that we would use to when developing the robot. Basically the same thing happened as if you would type the text "kapow" into the text field of Google in a normal browser.

Let us take a look at the generated Enter Value step. Click on the Enter Value step so you can see the configurations of that step in the right of RoboMaker. The step configuration consists of 4 tabs, let us take a quick look at each tab. For a more detailed look at exactly what configurations each step has please refer to the RoboMaker help and documentation. The first tab is the "Basic" tab, that contains the name and comment of the step. This is a perfect place to document the step by a breif description and a good name. Changing the comment is shown as a little document icon on the step and pops up when moving the mouse over the step.

 The second tab is the "Tag Finders" tab, where it is configured what HTML tags the step should interact with. In this case it is the tag path to the Google text field. In most cases there is no need to manually edit the tag finders, instead we can just automatically generate them as we have done here. But if we want to change the Tag Finders we have many options on how to do so, please see documentation and other tutorials for more information on this.

 Tab number 3 is "Action" which defines exactly what this step is going to do. In this case it will take the text from the attribute RESTInput.value1 (ie it will take the text "kapow" from "searchText") and enter this text into the tag that the Tag Finders found. Just as for the Tag Finders tab there is generally no need to manually edit this information, the generated settings are perfect in most cases.

 Finally we have the "Error Handling" tab in which we can set how errors should be handled if they occure. This step will be discussed in other tutorials and in this tutorial we are just assuming that the world is prefect and that everything works as it should at all times.

 All steps have the same "Basic", "Tag Finders" and "Error Handling" tabs, it is only the "Action" tab that differs from step to step. After entering the text we want to search for into the Google text field we need to make the robot click on the "Google Search" button. Once again make sure that the "Return Response" step is active (if not, just click on it). Then right click on the "Google Search" button in the browser view and choose Click.

Yet another step has now been added to the robot - "Click Google Search". This step does the same thing as if you in a browser yourself had clicked the button in a browser, which means that the browser view now shows the Google search result for the term "kapow".

 Now when the search is done it is time to get the title and the URL for the first result and return this data. To get the title of the first search result right click on the title in the browser view, choose "Extraction"-"Extract Text"-"RESTOutput" and finally "RESTOutput.value1 (title).

This will create a new step called "Extract Value 1" in our robot.

 This step takes the title of the first search result, extract it and move it into the attribute value1 of the output object RESTOutput. If wew look at the value of RESTOutput.value1 we see that is is "Kapow Technologies is a leader....". If we click on the "Extract Value 1" step we see that the RESTOutput.value1 becomes empty, then click once again at the "Return Response" step and see the value come back. This is another example of how RoboMaker executes every step as it goes along, even during development.

 Now do the same for the URL of the first search result. Right click on the same place as you did for extracting the title, but instead if choosing "Extract Text" take "Extract URL" into "RESTOutput.value2 (url)". This will be the final step we add to this robot in this tutorial.

This is a very simple robot and we can already be quite confident that it works, but to be safe we want to test it, so time to check out the RoboMaker debugger.

Part 3 - Test the robot

Compared to most development environments there is a lot of debugging going on as you do the acctual development. If a variable has the wrong data or if something do not work you will notice as RoboMaker execute each step during development. For the simple robot we have done in this tutorial there is probably not a need for much more testing than that, but for larger and more complicated you probably want to test that the robot can handle incorrect and missing input data etc. To do this we use the RoboDebugger which you open by clicking on the little blue bug in the RoboMaker icon bar.

 This opens the RoboDebugger in a new window and at the top of the window we se the steps of our REST robot. Here we can run the robot and see what output it produces using with different input values. Either we can run the whole robot or we can single-step through the robot. For each step we can see what values each object attribute has. We can also put in breakpoints so that the debugger stops at a specific step. Let's start by clicking the blue run arrow icon and see the robot be executed.

  Now we can see what result the robot would produce if the input value "searchText" was set to "kapow". Since this was the value we used while developing the robot we already knew it would work, it is more interesting to test the robot using some completely different value. To do this we change the value "kapow" to "robot" and then run the robot again, this time with completely different output. So the robot works even if we use some different input value than what we used to develop the robot, that is good stuff. In this tutorial we will not handle the case when Google does not find any search results, this is of course something that should be handled if this was going to be a robot to use in real life, but it is outside the scope of this tutorial. So let us for a moment live happily in a world where all input is valid.

Now the robot is tested and done, time to publish it to openkapow.com so we can use it.

Part 4 - Save, publish and run the robot

We have a pretty cool robot already, but as long as it is just local on your computer it is quite pointless. So let us publish it to openkapow.com so we can use the REST service. The first thing to do is to save the robot locally. This is not necessary to publish the robot to openkapow.com, but it is always a good idea to do so anyway. Just take "Save" from the "File" menu and save your robot file on your harddrive.

 Next thing to do is to publish the robot. This means that the robot is uploaded to openkapow.com and can be run from there using a simple URL in a browser or a program (for example in your PHP, JSP, ASP or Ruby on Rails code). Once a robot is uploaded to openkapow.com it is publically available to all and everyone to use or download.  In order to publish a robot you need to be registered as a user at openkapow.com and you need to have put in your openkapow username and password in RoboMaker. You probably did this when you installed RoboMaker already, if not simply go into the "File" menu and take "Edit Username and Password". When the username and password are correctly configured click the little openkapow robot icon to open the publish dialouge box.

The title and description of the robot are already filled in, based on what you entered when first creating the robot. Feel free to edit that information if you want to. Each robot needs to be categorized as beloning a category, in this case we take "Tutorials and Example" (but if you do a similar robot it should probably be in the category "Searching"). Each robot can also have tags associated with it. These tags makes it easier to find a robot when searching on openkapow.com, so add the tags you think best describe your robot.

Once we are done with adding tags and configuring our robot click publish. RoboMaker then contacts openkapow.com and uploads your robot. When this is done you will get back the URL through which the robot is now available to use. This is the URL to use to interact with the robot.

 If you open up that URL in a browser you will get to a screen where you can specify the input values to the robot, in this case there will only be one input value ("searchText") and it will have the default value we defined ("robot"). Feel free to change the value of searchText to whatever you want to search for at Google.

 Under the tab "Result format" you can specify in what format the REST robot should return it's results. This means that one robot can return it's output as either HTML, JSON, XML, XHTML or CSV. Test the different formats to see the difference.

When calling a REST robot it is of course not very common to go via these pages to set the input value and the result format of the robot. Instead one wants to call the REST robot directly with one URL that defines both input values and result format. This is easily done, simply call the robot with the input values and the result format as parameters in the URL. For example, using the robot we just have published to search for the text "kapow" and return the output data as XML we would use the URL:

http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=kapow&resultformat=xml

To instead search for "robot" and return the output as HTML we would use the URL:

http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=robot&resultformat=html

In the demo section there are examples of how to use REST robots directly from PHP, JSP, Javascript and Ruby On Rails to build mashups. This is done by calling the robots directly from some program code using the robots URL with different parameters.

Summary

We have with a few clicks of a mouse created a REST service that searches Google and returns the first search result. We haven't written any code at all, not even changed any of the automatically generated configurations. This is a good example how easy it is to develop robots and how quickly you will see results.

Of course it would be a much better robot if we could return all the search results from Google instead of just the first on, this is exactly what we are going to do in the next tutorial.

Published Friday, November 24, 2006 2:37 PM by Andreas

Comments

 

Tutorials said:

In this tutorial we will see how to create and test a REST robot that searches Google and returns the first 20 results.

November 27, 2006 3:32 AM
 

Tutorials said:

In this tutorial we will see how to create, publish and use a simple web clip robot. The robot will interact www.wired.com and just show the first three news items from that page in the clip.

November 27, 2006 3:43 AM
 

Tutorials said:

When learning a new programming language you will often start with a "Hello World" example,

November 29, 2006 2:47 AM
 

Tutorials said:

Here are answers to some Frequently Asked Questions about openkapow, robots and RoboMaker.

December 4, 2006 3:30 AM
 

Tips And Tricks said:

It can be a bit daunting to get started with building openkapow robots. On this site there is a lot of information such as tutorials, demos and of course all the robots to go through. But where do you start to master the Art of Robot Building in the quickest

December 7, 2006 4:46 AM
 

subhasreddy said:

I did this tutorial. Why is the output not in a clickable format? The url is just some text.

http://service.openkapow.com/subhasreddy/demo1googlesearch.rest. I thought that the search results would be clickable so that I can create some useful search robots.

But, if all it returns is a url in a text file, it doesn't make any sense. Please reply.

February 11, 2007 9:54 PM
 

Tutorials said:

In this tutorial we will see how to create, publish and use a simple web clip robot. The robot will interact www.wired.com and just show the first three news items from that page in the clip.

September 4, 2007 6:26 AM
Anonymous comments are disabled
Copyright 2006, 2007 KapowTech.com All Rights Reserved Company | Contact | Terms | Privacy