Tutorials

FAQ

2006-12-04T09:40:00Z

Here are answers to some Frequently Asked Questions about openkapow, robots and RoboMaker. If you do not find the answers to your questions here then please take a look in the forums.

Basics

Developing robots

Publishing robots

Running robots

If your questions are not answered by this FAQ then please take a look in the forums!

Basics

What is a robot?

An openkapow robot is a program created in RoboMaker. A robot interacts with web sites and automates actions on these sites that a person can do by hand in a normal browser (such as Internet Explorer or Firefox). This includes navigating web sites by clicking on links and submitting forms, extracting data from a site and much more. The behaviour of a robot can be affected by input values (for example the username and password to use to log in to a password protected site) and the robot produces an output (for example the current rate of a specific stock). There are three types of robots - RSS robots, REST robots and Web Clip robots. To start developing your own robots it is recommended that you start with the "Hello World" - My First Robot tutorial.

What is RoboMaker?

RoboMaker is the visual development environment used to create robots. In RoboMaker a robot is created by putting together steps and configuring those steps. This is done using a point-and-click interface that includes a browser view that allows the developer to see the page the robot interacts with in the same way as if it was in a normal browser such as Internet Explorer or Firefox. RoboMaker is a client program that is downloaded and run on the developers local machine. When a robot is complete it can be published to openkapow.com from within RoboMaker. Download RoboMaker here.

Why should I register at openkapow?

If you are a registered user of the openkapow community you can publish your own robots, download robots other users have made (to see how they work or improve them) and participate in the forums. Register now to become a part of the openkapow community.

What is RSS robot?

An RSS robot is a robot that creates an RSS feed or an Atom feed. The RSS feed can then be read using a special RSS Reader, a browser that supports RSS or a mail program that can read RSS. An RSS robot is run on a schedule defined by the robot developer. For example it can be run once an hour, which means that once an hour the feed will be updated with the new information the robot might have found. See Wikipedia for more information about RSS. To start developing your own RSS robots it is recommended that you start with the tutorial Creating a basic RSS robot that reads from Digg.

What is REST robot?
A REST robot is a robot that is run as a REST service. This means that it is called using a normal URL with all the input values as parameters (for example http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=kapow). The robot then outputs the result of it's work in for example XML, HTML or JSON. REST robots are normally used to create an API-like interaction with a web site and the REST robots are usually called from within a program (written in for example PHP, Ruby on Rails, C# or Java). This makes REST robots the perfect base for mashups. See Wikipedia for more information about REST. To start developing your own REST robots it is recommended that you start with the tutorial Creating a basic REST robot that searches Google.

What is a Web Clip robot?
A Web Clip is a reuse of a sites user interface. Using a Web Clip robot it is possible to take a complete site, a part of a site, or just a part of a page and use it outside the original site. Inside the robot it is possible to change the HTML, the stylesheets etc of the original site. Web Clip robots are many times used to take one piece of functionality from one web site and display it on another. To start developing your own Web Clip robots it is recommended that you start with the tutorial Creating a basic Web Clip robot that clips Wired News.

Back to the top of the page

Developing robots

What are steps?

Each step is one small piece of functionality (like clicking on a link or extracting the title of a page) and a robot is built up by several steps. The functionality of each step can be configured (for example to define what link to click on). Detailed information about all the steps available can be found in the help available inside RoboMaker.

What is a Tag Finder?
A Tag Finder is a part of the configuration of steps. It defines what HTML tag the step should interact with, for example what link to click on. The Tag Finders can be specified in a number of ways, for example using the DOM path of the HTML tag or using regular expressions (see the RoboMaker help for more information).

What type of robot should I choose?
That of course depends heavily what you want to do, what site you will interact with and how you want to use the robot, but here are some general pointers:

If you want to use a specific piece of functionality from a web site (for example searching stock symbols) and you just want to include this functionality on your own site then a Web Clip robot is usually a good choice.
If you want to use a web sites functionality or data from within your own programs then a REST robot is usually a good choice.
If you want to get information from a site and read this information using an RSS Reader than an RSS robot is usually a good choice.

To get an idea of what can be done with the different types of robots and what robot type fits your purposes you can take a look in the Robot Gallery to see what others have done.

Back to the top of the page

Publishing robots

How do I publish my robot?

When you are done developing your robot in RoboMaker and you are ready to publish it so that you can run then choose "Publish" from the File menu in RoboMaker. Once a robot is published you and everyone else can execute it. All registered users can also download the robot to see how it works and maybe do their own version (don't worry, they can't overwrite your version).

What should I call my robot?
You can call your robot whatever you want as long as it is a unique name, but it is recommended that everyone follows the same format so that robots are easier to find. When you are publishing your robot to openkapow.com from within RoboMaker you define both the Title and the URL of the robot. The Title should be something descriptive like "Add Todo to Basecamp REST". The URL should contain your username and a good URL name of the robot (sometimes shorter than the title, always without spaces and other special characters), for example "Andreas/addtodotobasecamprest". When the robot then is published on openkapow.com I can run the robot using the URL http://service.openkapow.com/Andreas/addtodotobasecamprest.rest.

Back to the top of the page

Running robots

How do I run a robot?

You run a robot by calling an URL that is specific for each robot, this URL is set when the robot is published on openkapow.com. If it is your own robot you can find the URL in RoboMaker in "Robot Configurations" found in the "File" menu, choose the tab "openkapow" in the "Robot Configurations" popup. For all published robots you can find the URL by clicking the "Try It" button on the robots page in the Robot Gallery. An example of such an URL is http://service.openkapow.com/tutorial/firstlinkgooglesearch.rest. Either a robot can be run from the web page that is shown when you load the robots URL into a browser, or the URL can be called directly from a RSS Reader (for RSS robots) or programs (for REST robots).

How do I get input to robots?
Assuming the robot you want to run acctually takes some input values then you can see exactly what input values it needs by trying the robot (use the "Try It" button found in the Robot Gallery). When you know what input values the robot needs you can call the robot with the input values as GET parameters in the URL. For example if a robot needs the input value "searchText" you can call it with the parameter ?searchText=kapow (http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=kapow).

What output formats does a robot have?
What output formats that are available for a robot depends what type of robot it is:

A Web Clip robots outputs a complete HTML page only.
An RSS robot can output data in several different RSS formats (0.91, 1.0 or 2.0) or using the Atom format. This is set by using the parameter "rssVersion" when calling the robot, for example http://service.openkapow.com/tutorial/rssdigginput?searchText=robot&rssVersion=atom_1.0. The default output format is RSS 0.91 if the "rssVersion" parameter is not specified.
A REST robot can output XML, JSON, CSV, HTML or XHTML. This is set by using the parameter "resultformat" when calling the robot, for example http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=kapow&resultformat=xml. The default result format of a robot is set in RoboMaker by the robot developer (File->Robot Configuration->Output formats) and if no "resultformat" paramter is specified the default result format is used.

Back to the top of the page

"Hello World" - My First Robot

2006-11-29T17:08:00Z

When learning a new programming language you will often start with a "Hello World" example, that will take you through some of the basic concepts of the language and the tools available. You can think of this tutorial as a "Hello Robot World" example, where the goal is to let you create, publish and use your very first openkapow robot. We will also cover the basic concepts of robot building and some of the tools available through the openkapow community. This tutorial goes through things on a very high level to give you a quick overview, you can find more tutorials that goes into much more depth in the Tutorials section.

Basic Concepts

Before starting you might want to familiarize you with a few of the basic concepts we use when we are talking about robot in the openkapow community.

RoboMaker is a visual development environment where you create robots that consists of steps working on the content of web pages. Robots can be run in RoboMaker and can be uploaded to the openkapow server where they are public available.

The rest of this tutorial requires that you have installed RoboMaker openkapow edition and registered as a user on the openkapow community site. If you have not then please Join our Community and download RoboMaker.

Creating a robot

You first robot will be a REST service robot. Normally REST robots will take some kind of input, process one or more web pages, and return an output in form of a XML document. For simplicity we will do without the input in this tutorial.

When starting up RoboMaker you will be met with a wizard guiding you through the first steps of creating a robot. For now you will need to choose the REST service robot type. When asked for start url for the robot you can type in "http://demo.openkapow.com/helloworld.html". Skip creating any input variables and create a single output variable named "myFirstOutput". This should give you screen similar to the one below.

[Click on the image to enlarge]

In this simple example we will assign the text in the gray box to the output variable named "myFirstOutput". To do this simply right click on the text "Hello Robot World" and select "Extraction", "extract text..." and then select "RESTOutput.value1 (myFirstOutput)".

[Click on the image to enlarge]

You now have a very basic robot that consists of only three simple steps:

In the first a web page is loaded.
The second step selects a piece of text on the page and assigns it to a field in the output object.
The third and final step returns the output object.

If you want you can try to run this robot in the build-in debugger be pressing the icon .

Saving and publishing a robot

Robots can be saved on you local hard disk where they will show up with a .robot extension. When you you feel you robot is ready to be deployed on the openkapow server you can publish it simply be pressing the icon. You will be asked to select a category and supply a few tags to help other users find you robots.

[Click on the image to enlarge]

After you have uploaded your robot you will see a link like http://service.openkapow.com/tutorial/helloworld.rest. The robot can now be executed from a normal browser or from within a program (in for example PHP, JSP, ASP, Javascript or Ruby on Rails). All robots published on openkapow are available for anyone to download and run, see for example the robot created in this tutorial.

What's next?

You can learn much more about creating robots in the the Tutorials section and read more about the basic concepts in Learn about robots. Good tutorials to get started with are:

Also take a look in the Robot Gallery to see what other people have created and check out the forums to get help and share ideas with other developers. To get an idea of what kind of mashups it is possible to build with openkapow you should take a look in the Demos section.

Creating a basic Web Clip robot that clips Wired news

2006-11-24T21:24:00Z

In this tutorial we will see how to create, publish and use a simple web clip robot. The robot will interact Wired and just show the first three news items from that page in the clip. The clip can then be used to show Wired's latest news on your own web site without using RSS feeds or any other technology other than openkapow. A web clip creates an interactive way of reusing the user interface of a site.

It is assumed that you already have downloaded the robot development environment RoboMaker and registered as a user on openkapow.com. It is also a good idea to have read the tutorials Creating a basic RSS robot that reads from Digg and Creating a basic REST robot that searches Google since these tutorials give a good overview of how RoboMaker works. All images in the tutorial can be viewed at full size, just click on the image you are interested in and it will be opened in a new window. The robot built in this tutorial is downloadable here

Part 1 - Create a Web Clip robot

Start with opening up RoboMaker, if you are using Windows you will find RoboMaker in your start menu under Kapow RoboSuite Openkapow Edition. In the startup wizard choose the option "Create a web clip..." and click OK.

In the New Robot Wizard that opens you need to enter the URL the clipping robot should start from, in this case we are going interact with Wired, so we enter "www.wired.com" and click Next.

We then need to specify if we want the clipping robot to login automatically. This is a very practical piece of functionality if we do a clipping robot that needs to login to a site show us the results. However, there is no need to login at Wired so we choose "No" and click Finish.

When we click the Finish button in the Wizard a whole host of things happen. First of all a 4 step robot is created, this is the most basic clipping robot possible. Second, the robot is executed. We can see this since the last step is active (shown by it being high lighted in green). The execution of the robot made a popup window called the "Portlet View" to open. In the Portlet View we can see what our clip would look like right now if we choose to publish it (you have already clipped the complete wired.com site - good job!). The Portlet View also contains some wizards that makes the whole process of building a clipping robot a lot simpler.

Let's take a closer look at the 4 initial steps before we go on with the robot building. These steps are specific to clipping robots and are not used in RSS or REST robots.

A clipping robot must start with a "Begin Clip" step, and a robot can have one, and only one, such step. It is the starting point of a clipping robot and it keeps track of the robots session and what page to load. The "Default?" step is the first step on the default branch of a clipping robot, more about this soon when we have added another branch (and if you do not know what a branch is yet, then don't worry, we will get to that as well). The "Clip" step is where it is defined exactly what in the page that should be clipped, the default is that the whole page is clipped (in this case the complete Wired.com startpage), but this can, as we will soon see, be easily changed. The "End Clip" step is (and must be) the last step in a branch of a clipping robot. It returns the data that the Clip step has clipped. It will become clear what these steps do as soon as we add some functionality to our robot, so let us turn our attention to the Portlet View to do just that.

If you at any point while developing a clipping robot have closed the Portlet View it can be easily opened by either clicking on the End Clip step in a branch or by clicking the icon in the icon bar of RoboMaker. As mentioned before the Portlet View shows what our clip would look like right now if we choose to publish it. But it does much more than that, it also gives us a way to interact with the loaded page and to add functionality to the clipping robot using several handy wizards.

Part 2 - Add a second branch and clip the top 3 news of Wired

Right now we are clipping the whole of the site, but what if we just want to clip the Wired logo and the top 3 news stories? We are not interested in the rest of the start page at all. However, for all other pages than the start page Wired.com we want to show the whole page, at least for now. To do that we open the Portlet View and click on the Add Clip Branch icon which will open the Add Clipping Branch Wizard.

In the first step of the wizard we need to give our new branch a name, for now, let's be happy with the default name "First Page" and click Next.

In the next step of the wizard we can define when this new clipping branch is going to be used, in this case it will be used when the page URL equals "http://www.wired.com". Perfect, no need to change that, so just click Finish.

We have now generated a new branch named "First Page" that contains three steps. Now we have a robot with two branches and it is the Clip step in the First Page branch that is active. This means that we now can change what that step will acctually clip.

Take a look on the Clip steps configuration to the right in RoboMaker. In the Tag Finders tab it is defined what the Clip step will clip, in this case everything on the whole page.

Let us change this and clip the first news story only. Start doing this by clicking on the title of the first news story in the browser view. Notice that when you do this the title is high lighted by a green dashed square, and the HTML tag you clicked on is also high lighted in the HTML source view, the DOM view and the HTML path. These four views are always connected which makes it easy to navigate a page.

Right now we do want to clip more than just the title of the first news story, we want to clip the whole first story. So we need to find the HTML tag that contains the story itself. Since we already have the title high lighted we can move out in the HTML from there until we find what we are looking for. Use the arrows in the browser view icon bar to do this . Using the arrows we move outwards in the HTML until we find the DIV tag with the class "borderBox" that holds the complete first story. Moving one step further out in the HTML will high light the DIV that contains all the stories, so then we know we have gone to far.

To clip the first story just right click on it in the browser view and choose "Use only this Tag".

This will replace the Clip steps Tag Finder that identified the whole page with a Tag Finder that identifies just this first news story.

Let us open the Portlet View and see how our clip has changed. Click on the "End Clip" step in the First Page branch and see for yourself.

With a few clicks we have change the content of the clip from the complete Wired.com page to just the first news story. That is nice, but we really wanted the first three stories, and for good measure, let's include the Wired logo as well. To do this we need to go back to the Clip step in the First Page branch and continue to configure it. Either close the Porlet View and click on the Clip step or click the "Edit Current Clip Branch" icon . to do all that in one go. Identify the tag that contains the second story in the same way as we did for the first story, then right click and choose "Use this Tag" to add it to the clip. The first time we took "Use only this Tag" to replace the Clip steps default Tag Finder that clips the whole page, but this time around we do not want to replace the existing Tag Finder (which clips the first news story), we just want to add more data to the clip. Hence we use "Use this Tag".

Do the same for the third news story and the Wired News logo. Then open the Portlet View and take a look at how your clip looks now. All the changes we have made to the clipping robot have been in the First Page branch, we have not touched the Default branch. We did configure the First Page branch to only be used when http://www.wired.com was loaded, so any other URL will be handled by the default branch. Test this by clicking on the link of the first news story inside the Portlet View. The URL will then be different from http://www.wired.com and thus the Default branch will be used. This branch will clip the whole page. Note that it is the End Clip in the Default branch that is active, which indicates that it was the default branch that was executed.

The way a clipping robot works is that it tries branch after branch, from top to bottom, to see which one to use to handle the current page that is being loaded. Only one branch will be used, and it will be the first one found who's rules fit the current page URL. The "First Page?" step contains a test if the URL equals "http://www.wired.com", if it does continue to the Clip step in that branch. If the "First Page?" step fails then the robot will try the "Default?" step in the next branch. The "Default?" step is a special type of step that is always true, so if no other branch is executed then the page will be handled by the Default branch.

So why did we add a new branch at all and not just edit the Clip step in the Default branch? Imagine if we didn't add a First Page branch, instead we clipped the top three news stories from Wired using the Clip step in the Default branch. This would work great for the Wired start page, but if we then clicked on a link in the clip (for example the link of the first news story as we did above) it would not look so great anymore. Since there is only one branch in the robot all pages will be handled by this one branch. Since the Clip step is set up to handle the Wired start page it will try to find the same data in any page, in this case it would try to find the top three news stories on a page containing a complete article, and it would of course fail. This is why the Default branch should always be keept untouched unless it is absolutely necessary. If you find yourself needing to edit the Default branch you should most likely add a new branch and edit the new branch instead.

The clipping robot we have so far shows the Wired logo and the top 3 news stories from Wired.com, and on all other pages it clips the whole page. We could of course continue to add branches to handle articles etc, but we leave that for the next tutorial. Instead let us take a look on how to publish the robot to openkapow.com and how to use it on our own pages.

Part 3 - Publish and use the Web Clip

We have a pretty cool robot already, but as long as it is just local on your computer it is quite pointless. So let us publish it to openkapow.com so we can use the clip to show Wired news on our own page. The first thing to do is to save the robot locally. This is not necessary to publish the robot to openkapow.com, but it is always a good idea to do so anyway. Just take "Save" from the "File" menu and save your robot file on your harddrive.

Next thing to do is to publish the robot. This means that the robot is uploaded to openkapow.com and can be run from there using a simple URL in a browser. Once a robot is uploaded to openkapow.com it is publically available to all to use or download. In order to publish a robot you need to be registered as a user at openkapow.com and you need to have put in your openkapow username and password in RoboMaker. You probably did this when you installed RoboMaker already, if not simply go into the "File" menu and take "Edit Username and Password". When the username and password are correctly configured click the little openkapow robot icon to open the publish dialouge box.

The title and description of the robot are already filled in, based on what you entered when first creating the robot. Feel free to edit that information if you want to. Each robot needs to be categorized as beloning a category, in this case we take "Tutorials and Examples" (but if you do this robot yourself you should publish your robot in the "News" category). Each robot can also have tags associated with it. These tags makes it easier to find a robot when searching on openkapow.com, so add the tags you think best describe your robot.

Once we are done with adding tags and configuring our robot click publish. RoboMaker then contacts openkapow.com and uploads your robot. When this is done you will get back the URL through which the robot is now available to use.

You (or anybody else) can now use this Web Clip robot directly from a browser by just using this URL (see for yourself at http://service.openkapow.com/tutorial/wired3stories.clip). For example this clip can be integrated to any existing page simply by loading the clip in a frame or iframe. The clip shown in the browser if fully usable and all links are clickable just as for any other page. If the page loaded is http://www.wired.com the top three stories will be shown, in all other cases the whole page will be shown. If you want to find the URL to one of your already published robots you can find that in "Robot Configuration" in the "File" menu in RoboMaker. The whole robot is downloadable here.

Summary

We have with a few clicks of a mouse created a clip containing the top stories from Wwired. We haven't written any code at all, not even changed any of the automatically generated configurations. This is a good example how easy it is to develop robots and how quickly you will see results. This clip could now be integrated with your page or your blog in a frame or an iframe for example. Basically you have with a few clicks reused and remixed Wired.com to look and do what you want you want it to. There have been no need to write any code whatsoever, and no need to install any servers or fiddle with deployment files. This is a very powerful way to reuse parts of pages, both content and functionality.

Creating a basic REST robot that searches Google

2006-11-24T20:37:00Z

In this tutorial we will see how to create, test, publish and use a very simple REST robot using openkapow. The robot will create a REST service that searches Google and returns the first result of the search. This tutorial describes the process in great detail and most of it applies to RSS robots as well, and in some degree also to Clipping robots.

It is assumed that you already have downloaded the robot development environment RoboMaker and registered as a user on openkapow. All images in the tutorial can be viewed at full size, just click on the image you are interested in and it will be opened in a new window. The robot built in this tutorial is downloadable here.

Part 1 - Create a REST robot

In the New Robot Wizard that opens you need to enter the URL the REST robot should start from, in this case we are going to search Google, so we enter "www.google.com" and click Next.

In the next screen of the New Robot Wizard we enter all the input values we want our robot to have. Since we want to search Google we need to know what to search for. This will be an input value to the robot. So write "searchText" in the input value text field and click "Add". This robot only need one input value, so when we have added "searchText", click Next.

On the next screen we add the default value for the "searchText" input value. This is the value we will use while developing the robot. In this case we will need a text to search for in order to get any results from Google. So let us use the term "kapow" during development. Once this robot is done and published to openkapow.com the input value "searchText" will not in any way be hardcoded to "kapow", instead this will be a parameter that anybody using the robot can specify in the url that calls the robot.

We have now defined the input to our REST robot, so it is time to define some output values. The robot should return the first search result from Google. For this first search result we want both the title and the URL, so add "title" and "url" as output values. Then click Finish to generate the new REST robot.

We have now created a very basic REST robot that contains two steps - one "Load Page" and one "Return Response" step - as you can see at the top of RoboMaker. The Load Page step, does just what the name implies, loads the a page, just like if you used a browser to do the same thing. The Return Response step returns data from the robot and this is the data that the REST service will return.

The robot will start out with the last step active, which is indicated by the green highlighting of the step. Click on the "Load Page" step to make it active. In the right of the screen are the settings for the current step, which now should be Load Page. As you can see the URL "www.google.com" is to be loaded in this step.

If you click on the "Return Item" step, all steps before that will be executed, in this case only the "Load Page" step. This means that www.google.com will be loaded into RoboMakers internal browser view. RoboMaker executes the steps during development in the same way as when a robot is run on openkapow.com. This feature is very usefull as you will always see what state the robot is on a particular step. In the right of the screen we can now see the the step configuration for the active step. In that configuration we see that the Return Response step will return an object called RESTOutput.

Let us take a look on the Input Objects in the lower right of the RoboMaker screen. Here we can see how the input and the output we defined in the New Robot Wizard are represented in the robot. The RESTInput object has 10 name-value pairs. In this case we only specified one input value, so the attribute "name1" is set to "searchText" and the attribute "value1" is set to "robot". This means that when somebody calls the robot via an openkapow.com URL using the parameter "searchText" (for example http://www.openkapow.com/....?searchText=kapow") then the value of that parameter will be available within the robot in the "value1" attribute. Note that if the robot has several input parameters the order does not matter, the parameter "searchText" will always be mapped to name1-value1 (in this case). Output Objects work the same way as input objects, and every REST robot uses the output object RESTOutput, which has 10 name-value pairs, just as RESTInput has. For this robot we see that RESTOutput.name1 is set to "title" and RESTOutput.url is set to "url", which are the output values we defined in the New Robot Wizard. The use of both RESTInput and RESTOutput will clarify as we continue this tutorial.

Part 2 - Search Google

We have now created a REST robot, so it is time to add some functionality to it. The robot should search Google and return the first result. So the robot needs to write the content of the input value "searchText" into the Google text field, then the robot should click the "Search" button on Google and finally the robot should read the first search result and return that data. First thing first, we need to make the robot enter the value of the input value "searchText" into the Google text field. To do this click on the Google text field in the browser view.

Notice that the title is highlighted in 4 different places, in the browser view, in the HTML source code, in the HTML path and in the DOM (don't worry if you do not know what all this means, you do not need to know this if you are not going to make much more complicated robots than this one). We have clicked on the "input" HTML tag which represents the search text input field. To make the robot enter the searchText into this text field simply right click on the text field in the browser view and then take "Enter Text from Attribute", choose the object "RESTInput" and the attribute "RESTInput.value1 (searchText)".

Several things have now happened. A new step called "Enter Value 1" have been inserted between Load Page and Return Response. If you look in the browser view you will see that the text field now contains the value "kapow", which we defined as the searchText that we would use to when developing the robot. Basically the same thing happened as if you would type the text "kapow" into the text field of Google in a normal browser.

Let us take a look at the generated Enter Value step. Click on the Enter Value step so you can see the configurations of that step in the right of RoboMaker. The step configuration consists of 4 tabs, let us take a quick look at each tab. For a more detailed look at exactly what configurations each step has please refer to the RoboMaker help and documentation. The first tab is the "Basic" tab, that contains the name and comment of the step. This is a perfect place to document the step by a breif description and a good name. Changing the comment is shown as a little document icon on the step and pops up when moving the mouse over the step.

The second tab is the "Tag Finders" tab, where it is configured what HTML tags the step should interact with. In this case it is the tag path to the Google text field. In most cases there is no need to manually edit the tag finders, instead we can just automatically generate them as we have done here. But if we want to change the Tag Finders we have many options on how to do so, please see documentation and other tutorials for more information on this.

Tab number 3 is "Action" which defines exactly what this step is going to do. In this case it will take the text from the attribute RESTInput.value1 (ie it will take the text "kapow" from "searchText") and enter this text into the tag that the Tag Finders found. Just as for the Tag Finders tab there is generally no need to manually edit this information, the generated settings are perfect in most cases.

Finally we have the "Error Handling" tab in which we can set how errors should be handled if they occure. This step will be discussed in other tutorials and in this tutorial we are just assuming that the world is prefect and that everything works as it should at all times.

All steps have the same "Basic", "Tag Finders" and "Error Handling" tabs, it is only the "Action" tab that differs from step to step. After entering the text we want to search for into the Google text field we need to make the robot click on the "Google Search" button. Once again make sure that the "Return Response" step is active (if not, just click on it). Then right click on the "Google Search" button in the browser view and choose Click.

Yet another step has now been added to the robot - "Click Google Search". This step does the same thing as if you in a browser yourself had clicked the button in a browser, which means that the browser view now shows the Google search result for the term "kapow".

Now when the search is done it is time to get the title and the URL for the first result and return this data. To get the title of the first search result right click on the title in the browser view, choose "Extraction"-"Extract Text"-"RESTOutput" and finally "RESTOutput.value1 (title).

This will create a new step called "Extract Value 1" in our robot.

This step takes the title of the first search result, extract it and move it into the attribute value1 of the output object RESTOutput. If wew look at the value of RESTOutput.value1 we see that is is "Kapow Technologies is a leader....". If we click on the "Extract Value 1" step we see that the RESTOutput.value1 becomes empty, then click once again at the "Return Response" step and see the value come back. This is another example of how RoboMaker executes every step as it goes along, even during development.

Now do the same for the URL of the first search result. Right click on the same place as you did for extracting the title, but instead if choosing "Extract Text" take "Extract URL" into "RESTOutput.value2 (url)". This will be the final step we add to this robot in this tutorial.

This is a very simple robot and we can already be quite confident that it works, but to be safe we want to test it, so time to check out the RoboMaker debugger.

Part 3 - Test the robot

Compared to most development environments there is a lot of debugging going on as you do the acctual development. If a variable has the wrong data or if something do not work you will notice as RoboMaker execute each step during development. For the simple robot we have done in this tutorial there is probably not a need for much more testing than that, but for larger and more complicated you probably want to test that the robot can handle incorrect and missing input data etc. To do this we use the RoboDebugger which you open by clicking on the little blue bug in the RoboMaker icon bar.

This opens the RoboDebugger in a new window and at the top of the window we se the steps of our REST robot. Here we can run the robot and see what output it produces using with different input values. Either we can run the whole robot or we can single-step through the robot. For each step we can see what values each object attribute has. We can also put in breakpoints so that the debugger stops at a specific step. Let's start by clicking the blue run arrow icon and see the robot be executed.

Now we can see what result the robot would produce if the input value "searchText" was set to "kapow". Since this was the value we used while developing the robot we already knew it would work, it is more interesting to test the robot using some completely different value. To do this we change the value "kapow" to "robot" and then run the robot again, this time with completely different output. So the robot works even if we use some different input value than what we used to develop the robot, that is good stuff. In this tutorial we will not handle the case when Google does not find any search results, this is of course something that should be handled if this was going to be a robot to use in real life, but it is outside the scope of this tutorial. So let us for a moment live happily in a world where all input is valid.

Now the robot is tested and done, time to publish it to openkapow.com so we can use it.

Part 4 - Save, publish and run the robot

We have a pretty cool robot already, but as long as it is just local on your computer it is quite pointless. So let us publish it to openkapow.com so we can use the REST service. The first thing to do is to save the robot locally. This is not necessary to publish the robot to openkapow.com, but it is always a good idea to do so anyway. Just take "Save" from the "File" menu and save your robot file on your harddrive.

Next thing to do is to publish the robot. This means that the robot is uploaded to openkapow.com and can be run from there using a simple URL in a browser or a program (for example in your PHP, JSP, ASP or Ruby on Rails code). Once a robot is uploaded to openkapow.com it is publically available to all and everyone to use or download. In order to publish a robot you need to be registered as a user at openkapow.com and you need to have put in your openkapow username and password in RoboMaker. You probably did this when you installed RoboMaker already, if not simply go into the "File" menu and take "Edit Username and Password". When the username and password are correctly configured click the little openkapow robot icon to open the publish dialouge box.

The title and description of the robot are already filled in, based on what you entered when first creating the robot. Feel free to edit that information if you want to. Each robot needs to be categorized as beloning a category, in this case we take "Tutorials and Example" (but if you do a similar robot it should probably be in the category "Searching"). Each robot can also have tags associated with it. These tags makes it easier to find a robot when searching on openkapow.com, so add the tags you think best describe your robot.

Once we are done with adding tags and configuring our robot click publish. RoboMaker then contacts openkapow.com and uploads your robot. When this is done you will get back the URL through which the robot is now available to use. This is the URL to use to interact with the robot.

If you open up that URL in a browser you will get to a screen where you can specify the input values to the robot, in this case there will only be one input value ("searchText") and it will have the default value we defined ("robot"). Feel free to change the value of searchText to whatever you want to search for at Google.

Under the tab "Result format" you can specify in what format the REST robot should return it's results. This means that one robot can return it's output as either HTML, JSON, XML, XHTML or CSV. Test the different formats to see the difference.

When calling a REST robot it is of course not very common to go via these pages to set the input value and the result format of the robot. Instead one wants to call the REST robot directly with one URL that defines both input values and result format. This is easily done, simply call the robot with the input values and the result format as parameters in the URL. For example, using the robot we just have published to search for the text "kapow" and return the output data as XML we would use the URL:

http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=kapow&resultformat=xml

To instead search for "robot" and return the output as HTML we would use the URL:

http://service.openkapow.com/tutorial/firstlinkgooglesearch?searchText=robot&resultformat=html

In the demo section there are examples of how to use REST robots directly from PHP, JSP, Javascript and Ruby On Rails to build mashups. This is done by calling the robots directly from some program code using the robots URL with different parameters.

Summary

We have with a few clicks of a mouse created a REST service that searches Google and returns the first search result. We haven't written any code at all, not even changed any of the automatically generated configurations. This is a good example how easy it is to develop robots and how quickly you will see results.

Of course it would be a much better robot if we could return all the search results from Google instead of just the first on, this is exactly what we are going to do in the next tutorial.

Improve Performance and Robustness of a robot

2006-11-24T13:16:00Z

In this tutorial we will see how to improve the performance and the robustness of an RSS robot. To do this we need to go into some of the more advanced and powerfull features of RoboMaker.

This tutorial builds on the other RSS tutorials about how to build a basic RSS robot, how to use For Each, Repeat-Next loops and how to use input values in the RSS robot. It is assumed that you have read and understood these tutorials and that you already have downloaded the robot development environment RoboMaker and registered as a user on openkapow. All images in the tutorial can be viewed at full size, just click on the image you are interested in and it will be opened in a new window. Of course the robot built in this tutorial is downloadable here.

Part 1 - Improving Robustness

A robust robot is a robot that will function even if the web site it interacts with changes. It is of course impossible to make a robot that can handle all thinkable and unthinkable scenarios, but a few easy changes can make a robot much more able to handle minor changes such as layout changes or added content. It is very uncommon that sites do a major rehaul of their design and structure, and it is definitly not more common than changes in other kinds of integration through API's, XML files etc.

As an example of improving the robustness of a robot we continue to work on the robot from the tutorial Creating an RSS robot with an input value to search Digg. If you haven't read that tutorial, now would be a good time. This robot searches Digg based on an input value to the robot. It returns an RSS feed based on the search results.

Most steps in this robot have been added by right clicking in the browser view in RoboMaker and then choose the type of step that should interact with the highlighted HTML tag. This is a very quick and convenient way of adding steps, and in the vast majority of cases the configuration of the steps (as created by RoboMaker) works perfectly. Sometimes though, it is good to add a bit more thought and care into the step configuration. Let us start by taking a closer look at the Tag Finder configuration of the "Test Tag" step.

The Tag Finder tries to find a div with the attribute name class and the value notice anywhere on the page. To use the attribute name class and the attribute value notice is very good for robustness since this points the step to just the correct div tag. What is not pefect for robustness in this step is the Tag Path .*.div.div.div. This Tag Path points to any div tag contained inside two other div tags. So what if Digg changes it's structure and the div we are looking for is no longer contained within two other div's? Then the step will not find what it is looking for, no matter if we can see the div in a browser. If we instead change the Tag Path manually to .*.div we immediatly improve the robustness of this step, because now the step is really looking for a div with the class notice anywhere on the page. Do not forget to test your robot in the RoboDebugger after each change, otherwise you run the risk of "improving" your robot so much that it acctually does not work.

Our robot is quite robust already thanks to the For Each loop we are using. This loop creates a Current Tag and moves that Current Tag in each iteration of the loop. All the Extract steps inside the loop have their Tag Finders defined in relationship to this Current Tag. If we take a look at the Tag Finders of the "Extract Title" step for example, we see that instead of this Tag Finder trying to find something anywhere on the page it tries to find something inside the Current Tag 1, which in this case is defined by the For Each step.

Using Current Tags is a very powerful way of making your robots more robust and if you are interacting with big complicated web pages it is also a very good way for yourself to get more control of what steps interact with what tags on the page. The For Each loops use Current Tags, but if you want to set a Current Tag ourside a loop then simply use the step "Set Current Tag". Once a Current Tag is set then all Tag Finders in subsequent steps can be set in relation to this Current Tag. If you define several Current Tags subsequent tag finders can even be defined in relation to all your Current Tags, for example as between Current Tag 1 and Current Tag 2. Let's use the Set Current Tag step to improve the robustness of the "Click Next" step. Currently the Tag Finder of the Click step is looking for a a tag with the class nextprev and that fits the pattern .*>Next.*, the problem is that the a tag has to be within three div tags.

To improve this Tag Finder, and to see how to use the Set Current Tag step, let's add a a Set Current Tag step in front of the Click next step. This is done by making the "Click Next" step active, identifiying the div tag that contains all the paging links on the search result page and then right clicking on that div and choose "Other" and "Set as Current Tag".

Now we have added an "Set Current Tag" step between "Test Tag" and "Click Next". The Tag Path in the Tag Finder of the new step is now set to ".*.div.div.div", change this to ".*.div" to improve the robustness of the step.

Time to change the Tag Finder in the "Click Next" step so that it uses the new Current Tag. Either you can do this manually by editing the values in the existing Tag Finder, or you can make the "Click Next" step active, right click in the browser view on the tag you want to use and choose "Use only this Tag" to replace the current Tag Finder. Beware if you do it the latter way you will have to reenter the needed Tag Pattern ".*>Next.*" in the Tag Finder. Instead of doing that, let's do it manually. Open of the Tag Finder of the "Click Next" step and change the Find Where to "In Current Tag". Set in this tag to "Current Tag 1" and change the Tag Path to be ".*.a".

The "Click Next" step is now much more robust since it does not matter where on the page the div with all the page links and the next/previous links are, it will still find it and click in the correct link. Using a "Set Current Tag" in this situation is a bit of an overkill, a much simpler way of doing things would be to just change the Tag Finder of the "Click Next" step (and not add any current tag step at all) to find anywhere on page and the tag path ".*.a". But if we did that you would not have seen the usefullness and power of the "Set Current Tag" step.

Part 2 - Improving Performance

Performance is of course very important in robots that will interact directly with a user, we do not want the user to sit and wait for seconds and seconds while our robot does it's thing. This makes performance less important for RSS robots compared to REST and Clipping robots. Especially since the RSS robots are run on a preset frequency and the result cached inbetween those executions. But nevertheless performance is important for RSS robots, and if nothing else, we can use our current Digg RSS robot as an example of how to improve performance of all types of robots.

Openkapow robots interact with web sites. Thus the performance of all robots depend on things like the response time of the web site it is interacting with, the available bandwidth and many other factors outside our control when we are building robots. As we will soon see there are many things within our power in RoboMaker to improve performance anyway. The main gain to peformance is to avoid needless page loads. A page load is either a "Page Load" step, a "Click" step or any other step that loads a page from a web site. Steps that are purly internal to RoboMaker (extracts, conditions etc) usually take a few milliseconds to execute, while a page load might take many seconds depending on the web site in question (among many other things).

In the Digg RSS robot we are working on now there are not really any unnecessary page loads. If we really need to improve the performance we could replace the first few steps - that loads Digg.com, fills in the search term and clicks on the search button - with one page load directly to "http://digg.com/search?s=kapow" (if we are searching for the term "kapow"). This cuts down the execution time, but we are immediatly making the robot less robust since we require this quite specific URL to work.

Let us see what we can do to improve the robots performance without the hassle to have to change the page loads around, right now we are not that desperate to cut down the execution time. Before we get started with this it would be good to have a very rough idea of how long the robot takes to execute. Then at least we would have an idea of if we are in fact improving the execution time or not with the changes we will do. To get this rough exection time we open up the RoboDebugger and run the robot with the input search text "robot.

When RoboDebugger is done we see in the lower status bar of the debugger that the total execution time was 40.7 seconds. Keep in mind that this is when the robot ran in the debugger, which is very much slower than running the robot when it is published to openkapow.com! All we need this value for right now is to have a rough time to compare the execution time to, so we have an idea if our improvements acctually are improvements. If we run the robot again we will get another value (in my case 38.9 seconds) so 40.7 seconds is not set in stone, it depends on such things as how busy the machine it is running on is, the available bandwidth, if Digg has the search already cached, how busy Digg is etc etc.

We now have a value to compare our progress to so let us go back to RoboMaker and open the Robot Configuration (available in the File menu). Then we click on the "Configure..." button to configure the default options of the robot. These are options used by each step that loads a page (ie "Load Page" and "Click"). For each of those steps we can define if they should use the default options (which we are about to configure) or if the step should use it's own options. To change this go to the action tab of a load page step and you can define if this step should use the default options or not.

For now we just assume that all steps will use the default options. If this proves not to be a good idea when we are testing the robot we can change this and configure the options of some steps individually if necessary. But let's start with the default options. There are 5 tabs in the options window - All Loading, Page Loading, JavaScript Execution, JavaScript Event Handlers and Logging. In this tutorial we are just going to touch on some of the functionality in the first 3 tabs, but it is well worth to read the RoboMaker help and the manual to get to know the other options available here.

In the "All Loading" tab there is one setting in particular that is interesting to us right now, the "Enable Cookies" one. Our robot is not really using cookies for anything, it is not logging in to any system and there is no need to save any session data in a cookie. But is the robot downloading any cookies anyway? If we load the Digg.com page in the robot (simply click on the "Enter Value" step) and check the "Cookies" tab, we see that loading Digg.com involves 3 different cookies.

One of the cookies track the Digg session and two of them are probably used by some advertising network to track the ads they display on Digg.com. None of these are of any interest to us to let's uncheck "Enable Cookies" in the default options. Now no cookies will be downloaded when we run our robot, and that should already have saved some execution time. A quick test in RoboDebugger confirms that the robot is still working as it should and that the execution time is now down to 37.4 seconds. This is almost too small a change to mean anything (maybe Digg simply responded quicker this time), but at least nothing broke.

If we move on to the "Page Loading" tab in the default configuration window we find another very basic and very interesting option, the "Load Frames" option. The default is that all frames are loaded, but is this really what is needed in this case?

In RoboMaker each frame is loaded into it's own seperate subtab under the "Windows" tab. Here it is clear that as soon as we go in to Digg.com two frames are loaded. One that contains all the content we need to interact with, and one that contains a banner. Since the robot does not need this banner we are probably safe with not loading any tabs, instead we just want to load the main window. So let's uncked "Load Frames" and thest the robot again in RoboDebugger.

Sidenote: to see a site using many, many frames simply make a robot that loads CNN.com. There you've got a perfect example where it really pays of not to load all those unnecessary frames.

This time the total execution time of the robot in RoboDebugger is down to 24.7 seconds, quite a significant improvement. What we have done by not loading the banner is to cut the number of page loads in half. For each page load (Load Page,Click Submit and multiple Click Next's) the robot is now only loading the main window, and never ever the banner frame.

It is well worth noting that the "Images to Load" option in the "Page Loading" tab is per default set to "None". This means that when the robot is executed none of the images on the page are downloaded to the server, already this is a huge improvement on performance, and one that we get for free. Only this option means that it is usually quicker to load a page using a robot than in a normal browser.

The "Page Loading" tab in the options window contains another very powerfull mechanism when it comes to improve performance - Page Changes. When a robot is executed the content of the pages it loads are all moved into the servers memory so that the robot easily and quickly can interact with this HTML. The more content that is moved into memory, the more content the robot has to interact with and the slower the robot. Page Changes is a way to reduce the amount of HTML that is loaded into memory and thereby improving performance. When we are interacting with Digg in our current robot we are not interested in the sidebar that lists all topics, lets users log in etc. Still this sidebar is loaded into memory on each page load. If we use Page Changes to remove this sidebar from each page load we might save some execution time. The sidebar is of course still returned by Digg, but then it is removed before the HTML Digg returns is loaded into memory, so for the robot it is just like if this sidebar never existed.

Page Changes are very powerfull and there are many ways to configure them. For now we keep it simple and use the same Page Changes for all pages in the robot. To add Page Changes choose "Same for All Pages" in the Page Change dropdown.

One of the most common page change converter is a "Replace Pattern", so let us add one of those by clicking the "+" and then configure the new converter as "Text Formatting" and "Replace Pattern".

Double click on the new "Replace Pattern" converter to configure it. In a "Replace Pattern" converter we can define what patterns to find (using regular BLOCKED EXPRESSION and what to replace those patterns with. To know what patterns we are looking for we need to look at the page HTML in RoboMaker. There we can find the div with the id "sidebar" contains a lot of things in the page that we are not interested in. Furthermore we see that there is nothing in the HTML after that div that we are interested in, so let us remove that whole div and everything after it in one "Replace Pattern" page change. The pattern is "(<div id="sidebar".*)" and we replace it with an empty string.

All Page Changes needs to be carefully tested in RoboDebugger, if the pattern is a bit wrong we might remove parts of the page that the robot needs to do what it is suppose to do. Also keep in mind that any Page Change risks making the robot less robust, in this case we are very vulnerable if Digg moves the sidebar div to the top of it's HTML. Testing the robot in RoboDebugger once again proves that our changes has not broken the robot. The execution time is now 25.4 seconds, a slight bit more than before, but not enough of a change to really mean anything. Page Changes are very usefull when a robot interacts with a big page, in this case the page is already quite small, so the change in execution time is not great (of course there is also some overhead to executing the Page Changes that needs to be concidered).

If we move on to the next tab in the default options window we get to "JavaScript Execution" where there are many ways to configure how javascript will be executed in the sites that the robot uses. In this simple Digg RSS robot we do not really use any javascript, unless the links we click on have some onClick javascript event handlers that does something important. Let's simply uncheck "Execute JavaScript" in this tab and test the robot in the debugger. If it still works as we want then there is no reason why we should let our robot spend any extra time executing needless javascript. Another bonus we get by not executing javascript is that no external javascript documents will be imported when executing the robot. In the header tag of the Digg HTML we see that a javascript file called "utils.js" is imported, and if we do not import this file the robot should be a bit faster.

Testing the robot in the debugger now once again shows that the robot works fine, and this time the execution time is down to 22.5 seconds. This is almost half of what we started with (40.7 seconds), which is quite remarkable since we have not made any changes to the robot steps, only to the overall robot configuration.

The performance of the robot can be improved changing what the step configuration and even removing some steps. Some examples of what can be done are:

Replace the first 3 steps with simple one Load Page that goes directly to the search result page of Digg
Remove the top "Find Tag" branch and instead add the own error handling "Ignore and Skip Branch" to several steps. After all it is not the best for performance to always test for something (ie "No results found") that will only be there in exceptional cases.
Do not use the step "Set Current Tag" to define the Tag Finder for the "Click Next" step, just change the Tag Path to ".*.a" in the Click step instead (as discussed earlier in this tutorial).

To take a look in detail on this robot and to test it out you can download it here.

Summary

In this tutorial we have both improved the robustness and the performance of a RSS robot that interacts with Digg.com. We have talked about how to improve Tag Finders, how to use Current Tags and how to use the powerfull options in the Robot Configuration. Most of the topics in this tutorial works the same for REST and Clipping robots as for RSS robots and understanding how to make a robot more robust and how to improve performance also gives a better understanding of how openkapow and RoboMaker works. There are many options and configurations for each robot and for each step, and any given problem can usually be solved in a number of different ways. After this tutorial you should have a very good understanding of the potential of openkapow and you should be more than ready to build your own, quite complex robots with good robustness and performance.