tag:blogger.com,1999:blog-736810000699453506.post4719838889112446003..comments2023-05-11T00:49:36.314-07:00Comments on Ruminations on Computing: When to Test Manually and When to AutomateSteve Rowehttp://www.blogger.com/profile/17905356014908630180noreply@blogger.comBlogger12125tag:blogger.com,1999:blog-736810000699453506.post-60492111534436463482008-02-26T17:27:50.000-08:002008-02-26T17:27:50.000-08:00Hi Steve,Manual or automation is tricky decision, ...Hi Steve,<br>Manual or automation is tricky decision, if not taken carefully chances are high that you will fail in both. I suggest having mix of both as per project requirements. Manual testing for the modules that don't require regular changes and automation for frequently updating modules.Vijayhttp://www.softwaretestinghelp.com/10-tips-you-should-read-before-automating-your-testing-work/noreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-32766660144238927792008-02-26T18:40:58.000-08:002008-02-26T18:40:58.000-08:00Hi,about "Finding bugs right away while devel...Hi,<br>about "Finding bugs right away while development still has the code in their minds is best.", isn't that the aim of TDD?<br>What do you think about this? It seems that you place the testing phase after the developing one, not considering the "mixed" way. I will appreciate your guru :) response.<br>Have a good daycrazyTesternoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-49627482929786136442008-02-26T19:41:07.000-08:002008-02-26T19:41:07.000-08:00This is so nice overview that I would like to sugg...This is so nice overview that I would like to suggest some extension from my experience:<br>There is one benefit of automated tests I don't see mentioned - the regression bugs will be detected immediately by them while "development still has the code in their minds" - and so they know which change has caused the regression.<br>The benefit “Manual testing is the most flexible” also means that manual tester could change validation approach as they see something “suspicious” even during regression tests (warnings in log file, blinking and lagging UI items, etc., etc.). Automated will only validate what they are programmed to validate. <br>This is why when deciding what to automate I also take into account risk (probability and impact:fixing cost) of regression defects and how “smart” the validation should be.Ainars Galvanshttp://www.testingreflections.com/user/view/1469noreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-79931252463884902012008-02-26T23:56:16.000-08:002008-02-26T23:56:16.000-08:00Interesting post. A few questions;1. When using a ...Interesting post. A few questions;<br>1. When using a mixture of both, do you have separate resource for each? E.g. a group of testers doing the manual testing and a group of people owning the automation?<br>2. Does the effort involved in reviewing the automated tests ever become too high? E.g. in the time spent that 1 person could have been adding more value by running exploratory tests.<br>3. For anyone - Do people who dislike scripted tests still use automation? <br>I've always wanted to use automated testing but in the past we've never quite got it right - either it's become too much of an overhead or we lacked the resource to keep it going. I'm still searching for the right opportunity to use it.Simon Godfreynoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-36555105344095508552008-02-27T06:52:28.000-08:002008-02-27T06:52:28.000-08:00Lots of good questions. A few answers.@Vijay, eve...Lots of good questions. A few answers.<br>@Vijay, even for modules that don't change often, you still have to test them every release because the environment around them will have changed. Unless they are the lowest object on the stack, any changes below them means you need to retest.<br>@crazy - TDD is only a partial solution. It tests that each function or perhaps module works as expected but not that the whole system works. Many bugs are in the seams between modules.<br>@Simon:<br>1. I've seen it done both ways. There are advantages to both sides. I'm partial to separate resources but the danger is that they won't coordinate.<br>2. The cost of reviewing automated tests can become very high. However, I don't know that the time spent would be better used running exploratory tests. That would only be true if the automated tests were finding no issues. If that's the case, automating the review can be done easily.<br>To your last question, make sure you aren't automating too early. If the costs of automation are too high, there is likely too much churn in the project still. Either write tests that insulate you from the churn or wait until it has died down some.SteveRowenoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-30384547258747364832008-02-27T17:04:19.000-08:002008-02-27T17:04:19.000-08:00I keep wondering why manual vs. automation is view...I keep wondering why manual vs. automation is viewed as opposites and that tests can be performed either manually or can be automated.<br>Why not blend the two as part of the same test? Automation that assists testers with specific testing tasks while leaving other tasks that are part of the same test can often bring more value at a lower cost than trying to automate all or most parts of a test.<br>Developers use many tools to assist them in writing, managing, compiling, and building code. There's no reason that testing tools can't be created to help testers do their jobs better instead of trying to replace them.<br>Also, I can think of many cases in which disposable automation has helped me test something better or quicker than I could have done without the support of the automation. Automation does not always need to be run many times to become cost effective.<br>Ben Simo<br>http://www.QuestioningSoftware.comBen Simohttp://www.questioningsoftware.comnoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-56596782668286516892008-02-27T18:17:48.000-08:002008-02-27T18:17:48.000-08:00>>>The question then becomes how to decid...>>>The question then becomes how to decide what to automate and what to test manually. <br>When faced with this "when" and "what" about automation, I use following to help me out.<br>1. Tell me about the test - Is it ready, meaning documented? in what format? Studying and Analyzing the test can give lots of insight into the approach that I could take.<br>2. When I am getting into this kind of questioning? Is it before I design/document the test or at the time of test execution?<br>(Assuming that design and execution of tests are separate activities)<br>3. What kind of test is this? What kinds of oracles does it need?<br>Type of test -- I can think of a confirmatory test say data validation (this field takes only a 6 digit number (0-9)) or a business rule validation or a file comparison and so on ...<br>4. Does this require human to watch and observe when the test is being run? Is human intervention mandatory or optional?<br>5. Finally - I would ask who is best *agent* to *execute* this test ...<br>The notion of "best" would cover aspects like reliability, cost effectiveness, effort, time, chance of human errors and so on ...<br>There are some tasks that machine can do better and at some others human excels. That is key point for me determining what I should automate and what I should test manually....<br>Shrini Kulkarni<br>http://shrinik.blogspot.comShrininoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-33018470059070288962008-02-27T18:27:44.000-08:002008-02-27T18:27:44.000-08:00Ben Simo said : I keep wondering why manual vs. au...Ben Simo said : I keep wondering why manual vs. automation is viewed as opposites and that tests can be performed either manually or can be automated.<br>Let us classify tests in following categories:<br>A- Tests that ONLY Automation can execute<br>B- Tests that ONLY human can execute<br>C- Tests that BOTH human and automation execute with equal efficiency and effectiveness. Either can do the job.<br>D- Tests that BOTH human and automation can execute but there are situations where one *agent* scores over the other. Context is key.<br>E- Tests that human CAN execute but the accuracy, precision and result tend to diminish over repeated cycles - human fatigue causes "oversight" errors<br>F - Tests that automation CAN execute but at higher cost (human execution is cheaper and quicker option)<br>G- ??<br>So take a test and see which category it fits. That will give you clues about which agent is best suited to do the job ...<br>ShriniShrininoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-73817421611193122472008-02-27T18:51:57.000-08:002008-02-27T18:51:57.000-08:00>>>With a good test system, running the t...>>>With a good test system, running the tests becomes nearly free.<br>What do you mean by a good test system? running tests become free under what circumstances? What parameters make this happen and what parameters threaten this "free" service? Will the test execution be free (assuing that you can good test system) when the system is buggy, and undergoing lots of changes?<br>Also you would agree that by automation, you would also loose information/possibly bugs that might come from skilled human testing. Since with automation, you do not invest time <br>manual testing - hence free automated testing comes at a hidden risk .. how do you handle that?<br>ShriniShrininoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-81260489065505383192008-02-29T02:13:14.000-08:002008-02-29T02:13:14.000-08:00Charles Sterling on Rosario Specifications for som...Charles Sterling on Rosario Specifications for some of the TFS Features and Radio TFS!. Jason Barile...Team System Newshttp://teamsystemrocks.com/blogs/team_system_news/archive/2008/02/29/6247.aspxnoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-15824411083630616922008-06-02T17:12:07.000-07:002008-06-02T17:12:07.000-07:00Testing started simply. Developers would run their...Testing started simply. Developers would run their code after they wrote it to make sure it worked. WhenSteve Rowe's Bloghttp://blogs.msdn.com/steverowe/archive/2008/05/30/we-need-a-better-way-to-test.aspxnoreply@blogger.comtag:blogger.com,1999:blog-736810000699453506.post-49948120686785462092016-04-19T00:22:12.000-07:002016-04-19T00:22:12.000-07:00Hi Steve,If we go for automation tool like seleniu...Hi Steve,<br>If we go for automation tool like selenium it could be less cost than manual testing right?Balunoreply@blogger.com