Running a test suite with over a million test cases

We have a distributed test environment with the robotremoteserver starting a bunch of other applications and working with them as part of the test. The test that I am trying to run requires me to run over a million test cases in a single suite. The issue here is, when the pybot starts running, it blocks up 8 GB of RAM and results in slow performance. After a while it freezes up.

To overcome this issue I was planning to create separate suites with less than 100 cases in each. But in this case I am not able to use the keywords from the other remoteservers initiated from other suites. The only way I can use it is by disconnecting from the remoteserver and reconnecting it in each suite - which would defeat the purpose of the test.

I am not sure if anybody has come across this scenario. I would appreciate if anybody can think of a solution to this issue.

Additional information

Another thing that helps solve this: Is it is possible to call a keyword from a library (with state) that is initiated by another suite?

Can I use get library instance when using RIDE? I am importing the library in one suite, then try to get the library instance in other suite. Is this possible?

I keep getting the following error:

> AttributeError: Remote instance has no attribute 'replace'

In Suite 1 this is what I am doing:

Import Library    Remote    ${verifix_xmlrpc_url}    WITH NAME    Verifix
${lib}=    BuiltIn.Get Library Instance    Verifix
Set Global Variable    ${lib}

In Suite 2:

${lib}.remove messages    ${VenueNSDQ}

I don't know any specifics about Python, but you might want to try spinning up each test case as a separate process. This will allow you to recover the memory from each process as it completes, and may allow you to run your million test cases safely on one machine.

Generally, when a program needs more memory from the OS, it takes it, but it cannot give it back until it exits. This is likely why your process falls over.

Quick processes that you can spin up and then kill, will alleviate this memory problem.... and then it is a pretty short step from there to running each of these test cases through rsh on a different machine entirely.

I hope that helps a little.

OK. First, I do not program in Python, and I also have no idea what kind of computers you are using or how much memory the computer has or anything else about your situation with your computer(s). So this is just general information gleaned from writing system programs and working across multiple computers.

With that said: If you are going to run huge datasets or a huge number of programs you will have to partition up what you want to do so it will fit into the system(s) that you are using. That is the first part.

The second part is that if you want to talk across programs to other programs or systems then you need to set up some type of common memory usage. I know that PHP has such a thing built in to it so I have to assume that Python also has such a thing. You set up the common memory area and then send information back and forth between the various programs. Usually using some form of tokens to identify each program and/or system. In this way, you can set things up so once one suite of tests completes, it can tell the next set to begin and thus not take over or bog down your system.

If you are really good - you could also create some kind of controller program that starts up a test, watches it until the test ends/exits, and then start the next test. (So instead of a daisy chain way of doing this you have a master/slave way of doing this just like a client/server relationship.)

If you are using a Windows box/OS I would suggest looking at AutoIt. AutoIt can easily handle watching a program run and starting up a new task once an old task has finished. It also allows you full control over both your system as well as the remote system and was developed to help automate tasks such as the one you are trying to do. I found (by experimenting) that AutoIt can also be used with Unix/Linux boxes such as Macs and others. Although you only then have the commands you send over rather than access to the Windows features.

If you are more proficient with Python that you can do the above using it - then more power to you. Even though it has been two years since you asked this - I hope this helps you out in your endeavors. :-)