|
Tentative candidates: (not yet finalized) WaveSurfers: * CassiusClay 1.9.9.96 - expert WaveSurfer * Cyanide 1.80.b - expert WaveSurfer * Tigger 0.0.23 - good WaveSurfer, unique implementation * Chalk 1.2 - "fledgling" WaveSurfer :) Non-surfers: * Cigaret 1.31 - random orbital movement * GrubbmGrb 1.1.3 - MultiMode with Stop 'n' Go, Oscillation * RandomMovementBot - fluid, random movement * Butterfly - MultiMode; non-firing (prior?) version of Ugluk * FloodMini 1.4 - Kawigi's SandboxFlattener movement * DuelistMicro 1.22 - simple GoTo? movement, quite good against simple aimers Bots: TC2K6_reference_bots.zip (291 kb) |
|
/ReferenceBots? - /HowTo? - /FastLearning? - /Results? - /ResultsFastLearning? - /ResultsChat? - /PreChat? |
|
*Tango coughs* Well, I'm not the most qualified to comment on how to benchmark a gun, but I think CassiusClay and SandboxDT 1.91 would be the first two I'd nominate. CassiusClay is a great WaveSurfer that executes pretty quickly, and DT 1.91 is some really good RandomMovement. I'm working on the gun primarily for Dookious now, so I may have some better suggestions once I make some progress there. :) -- Voidious The problem is that there are not many good distinctive movements. We can throw in some more WaveSurfers, as they are the top-movers nowadays. Some special versions have to be made though, because SandboxDT and Tron/Shadow are the only ones I know of that you can set in challenge/reference mode without altering the code. -- GrubbmGait Well, it's true that nearly all the best movers are WaveSurfers these days, but getting to the top of the rankings also includes really smacking down every other type of movement, too. Perhaps an even mix of WaveSurfers and other movements? I think filling half the slots with WaveSurfers would be no problem, as many are open source - there's CassiusClay, SilverSurfer, RaikoMX, PowerHouse, Chalk, Dookious, Pugilist... (I could make the special versions, that's no problem.) PulsarMax also has a .properties file, and you mentioned Shadow. As for the non-surfers... We could keep DT 1.91 and Cigaret 1.31 from the first TC, as they generally got the lowest scores against them; MakoHT from the first TC had low scores, too. Doesn't GrubbmGrb have some pretty good movement for a non-surfer? Well, I guess we could look through the OpenSource by Movement type list to get some more ideas. -- Voidious I can think of a few other interesting, open source movements: *Tigger - wave surfing implemented with Tile Coding (a little slow, however) *Tityus - strong flat/random mover *Raiko 0.43 - another strong flat/random mover I second the vote for GrubbmGrb. He's the undisputed master of multiple choice strategies (and one of my favorite bots). Regarding Chalk, I'd be more than willing to make a TC version; though there's nothing particularily unique about his movement. -- Corbos I'll take a look at those tanks you listed when I get home, Corbos, but a couple thoughts for now... First, I really think using tanks that are fairly quick is a very good thing, and I think the overall quality of the challenge wouldn't be compromised by excluding tanks that are much slower. Also, I tend to think that two or three of the WaveSurfers should be among the most excellent surfers, and the others could be unique versions or just slightly closer to the middle of the pack. -- Voidious Oh, and I started a tentative list at the top of the page with the two candidates I feel most strongly about ;) Feel free to add to it. -- Voidious I modified and trimmed down Ugluk for an adaptive movement entry, pedersen.Butterfly, that you may find useful. As it is hit it cycles through various movement systems until it decides on the best one to use. If the initial choice gets beaten badly enough it will switch again. It is similar to the Musashi Trick, but more advanced. -- Martin Alan Pedersen A TC variant of GrubbmGrb is easily made, only set the 'MC_flag' and recompile. I've tried pedersen.Butterfly and it seems quite unique and versatile. One thing has to be solved though, it is not allowed to intentionaly ram disabled opponents. -- GrubbmGait Well, a "movement challenge" mode really shouldn't do that, but it actually shouldn't throw off the scores of a targeting challenge, right? The TC only measures bullet damage... -- Voidious @Voidious: I agree with your comments about SlowBots. Still, you've got to admit Tile Coding is cool! :) --Corbos Well, it certainly sounds cool, and I've wondered about it each time I've read the description ;) Now I simply have to read up on it, I guess! -- Voidious I've tossed the source code for pedersen.Butterfly (kinda) but I can recreate it. It just won't be as 'small' of code size. I'd think ramming would be a boon since it would end matches faster once they were unwinnable and give a better spread on the scores against it, but I can disable the ramming if it is beneficial to the challenge. -- Martin Alan Pedersen Heh, I thought the same thing about the ramming speeding things up... I guess it wouldn't make a huge difference when it's running minimized, though. The ramming shouldn't affect TC scores, but it would greatly affect any MC scores that would help guage the movement of that bot in a MovementChallenge. I'm not sure what I think of your Butterfly as a TC candidate, as Dookious got a near-perfect score against it :-\, but I would be interested to know how other tanks do against it first. -- Voidious I didn't expect Dookious to crush the bot, since you don't annihilate me in the Rumble (though you do win by better than 60%). If it's not challenging enough then you can scrap it. -- Martin You are right about the ramming, it does not influence the score. Ramming is not allowed in the movement-challenge though, therefor my (wrongly) remark. The naming of my flags is based upon the fact that normally I am a contender, not a reference-bot. At first GrubbmGrb could nail Butterfly easily, but after a round of 10-15 it could hardly hit it anymore. -- GrubbmGait |
|
One thing we should keep in mind: a good TC will benefit all Robocoders. A novice might get frustrated if they don't see improvement against bots like Shadow and CassiusClay (and let's face it, improvement is a slow process against these monsters). I haven't tried Butterfly but it could provide a strong challenge that isn't as intimidating as the top-10. A wide range of styles and rankings would be nice. Thoughts? --Corbos |
|
Welcome to the TargetingChallenge2K6. It's an attempt at measuring pure gun performance in 2006 :). The rules are simple. |
|
Yes, it is a good idea to represent various rankings in the TC. After all, your gun needs to be good against everyone, not just the top bots, to excel in the RR. --wcsv |
|
* The Challenger must remain stationary all the time. * The Challenger must try to fire power 3 bullets when it can. ** Which means no energy management whatsoever... ** Challenger is only allowed to feed the fire() method(s) the constant 3. This often results in at least one bullet per round fired with energy lower than 3 though. * There are 10 /ReferenceBots?, none of which fires, few of which does any ramming. * The Challenger meet each reference bot in battle over 500 rounds. * The Challenger's score against each reference bot is calculated like "challenger_bullet_damage / rounds" ** Average the scores against each reference bot to get the total score. |
|
Yep, I definitely agree. I guess I got the idea in my head that this challenge should be a lot *harder* (lower average scores) than the first one, but the truth is that it should be a consistent and accurate benchmark, first and foremost. |
|
Please post your bots' results on the /Results? page. This is made very easy by using RoboLeague and the TC Calculator. |
|
I still have a lot of trouble getting good scores against Cigaret 1.31, from the original TargetingChallenge, so I think that's another non-surfer that might be good to add. What do you guys think of the updated list of bots at the top? And what do you think of the the "2-3 superb surfers, 2-3 unique surfers, 5 non-surfers" distribution? As good as Toad's MC score is, he's kind of slow to execute, so I might want to skip him on the surfer selection. -- Voidious |
|
Results chatting is best carried out on the /ResultsChat? page. |
|
If no one objects, why don't we ask Voidious to finish the list and set things up? That's probably the best way to avoid the fates of TargetingChallenge2K4 and TargetingChallenge2K5. ;) --Corbos @Voidious: Do you have the time and are you willing to set up the TargetingChallenge2K6? If there is anything I can do to support you, please let me know (except bringing coffee). -- GrubbmGait Hey guys, I'd be honored to put together the list for TC2K6. I'll try and have it together within the next week or so, and hopefully you will all approve of it. :) I'm thinking 2 top-notch surfers, 3 good or unique surfers, and 5 non-surfers. I'll be sure to test sufficiently to try and get it as well-rounded as possible. -- Voidious |
|
If you're curious about how we arrived at this rule set, we took it from the original TargetingChallenge. If you're curious about how we arrived at this bot set, check out the /PreChat? page for the discussion. |
|
If anyone's interested, I will upload some of my RoboLeague test results to: https://www.dijitari.com/void/robocode/tc2k6 I put up the results of my first test just now. I'm going to use a dev Dookious, whose gun is pretty good against WaveSurfers, and a couple other guns to find the WaveSurfers. (This "dev" Dookious is just 0.61 plus another anti-surfing gun in the VG array.) Anyway, I won't comment with every update, but I will upload results there as they come about if anybody's curious. For reference, this gun gets about 89.5% in the original TargetingChallenge. Luckily, all of these bot authors have relatively clean code, so it was easy to get this setup quickly. I've ruled out Dookious as a candidate for now, as I'm a little biased, of course, and it seemed odd to test his movement against his own gun. And if anyone objects to their tank being a reference, just give a holler. -- Voidious I pit Ugluk vs. Ugluk on occasion. Ugluk's bearing offset gun is best against Ugluk's movement, which I think is best, with an 18% hit rate, which I'd prefer to reduce. -- Martin Well, guys, I got a tentative list going sooner than expected. It's at the top of the page. (All these TC bots can be found here.) Also, I've uploaded the Excel spreadsheet I was using to tabulate my test results, and the individual RoboLeague files are in that same directory. I want to run some more tests and contemplate the bot list some more before finalizing it, but I feel pretty good about this set of bots. For the WaveSurfers, part of my selection process was their execution speed; Shadow was about 60% as fast as Cyanide and CassiusClay, and Dookious was a bit slower than that. Tigger turned out to be pretty quick once I completely disabled his gun. Let me know what you guys think, or if you don't want your tank as part of the reference set. -- Voidious It's just my opinion; but I'd say set it up, post the bots, and let's get busy. Thanks Voidious. --Corbos I'll give it at least until the end of today for anyone to object, and I'll try and make sure there's nothing wrong with this set of bots. Hopefully, by tomorrow we'll have a new TargetingChallenge! -- Voidious Ok, I ran some tests to examine the accuracy of the new TC's test scores with this set of bots. Take a look, the guns are along the top: |500 rounds |Raiko 0.43 |Dookious 0.611|CC 1.9996|Shadow 3.64 |CassiusClay |59.412 |67.708 |57.062|69.042 |RaikoMX |51.146 |81.466 |68.246|71.896 |Chalk |78.140 |88.190 |94.068|91.954 |Cyanide |68.004 |78.598 |78.280|69.340 |Tigger |74.982 |84.274 |80.858|85.344 |Cigaret |84.636 |78.958 |89.714|83.716 |GrubbmGrb |91.012 |85.890 |92.458|91.264 |Butterfly |98.848 |99.814 |99.390|98.794 |RandomMovementBot |90.760 |87.246 |92.924|87.504 |NanoSatan Mu |94.436 |94.818 |94.818|93.732 |Score |79.138 |84.696 |84.782|84.259 |Original TC | 92.47 | 89.5 | 94.19 | ? Clearly, both Raiko and CassiusClay do a lot better against non-surfers than Dookious, while Dookious does significantly better against WaveSurfers. The RoboRumble clearly gives a lot more rating points to a gun that's good against non-surfers. (Edit: Due to the volume, I mean, not to imply that Dookious is as good or better than CC and Raiko!) Now, personally, I think that the above set of bots is a good benchmark of a gun's actual ability; but it clearly doesn't reflect the rating a gun would give in the RoboRumble. What do you guys think? Is the above a reasonable result, with the understanding that the non-surfer scores will affect your RR rating more? Or should there be less WaveSurfers in the reference set? -- Voidious I added Shadow 3.64's scores to the table above. I'm really thinking that the emphasis on anti-surfer targeting is a bit too strong, and that maybe I should remove RaikoMX and/or Cyanide, and add some more distinctive non-surfing movements. Or maybe we just count the non-surfing scores double? I dunno! I'd appreciate any input. -- Voidious Ok, I think replacing RaikoMX with FloodMini gives a more balanced view, and Kawigi's SandboxFlattener is near the top of the list of movements that I considered but initially didn't put in. Here's a look at the new scores, and I think it seems a bit closer to "reality" in the RoboRumble: |500 rounds|Raiko 0.43|Dookious 0.611|CC 1.9996|Shadow 3.64|LinearTargeting |CassiusClay|59.412|67.708|57.062|69.042|3.592 |Chalk|78.140|88.190|94.068|91.954|11.920 |Cyanide|68.004|78.598|78.280|69.340|3.592 |Tigger|74.982|84.274|80.858|85.344|30.544 |Anti-Surfer Avg|70.135|79.693|77.567|78.920|12.412 | | | | | | |FloodMini|89.472|89.478|93.708|87.186|70.044 |Cigaret|84.636|78.958|89.714|83.716|56.506 |GrubbmGrb|91.012|85.890|92.458|91.264|5.920 |Butterfly|98.848|99.814|99.390|98.794|34.048 |RandomMovementBot|90.760|87.246|92.924|87.504|86.058 |DuelistMicro|94.924|92.084|96.65|94.752|67.236 |Non-surfer Avg|91.609|88.912|94.141|90.536|53.302 | | | | | | |Unweighted Score|83.019|85.224|87.511|85.890|36.946 |Weighted Score*|84.451|85.839|88.616|86.664|39.672 The weighted score is (((2 * non-surfer avg) + surfer avg) / 3), and was just included for reference; I don't think it's necessary. What do you guys think of this list? The edited reference bots can be found in TC2K6_beta2.zip (287 kb). -- Voidious Looks great to me. FloodMini is a nice addition to the squad; its flat movement should give some good GF guns a run for their money. By the way, Thanks for setting this up , Voidious. It would take me a *very* long time to run the test matches that you did. --wcsv Great work Voidious! Looks very good to me. -- ABC NanoSatan Mu has a problem, after approx 200 rounds it generates a ArrayIndexOutOfBoundsException? at line 37. Either the source has to be adapted to eliminate the use of that 'ev'-variable, or another bot has to be chosen. I do like it simple but not so simple to hit movement though. Also how fast you kill it is important, because it hits the wall quite often. The choice for FloodMini is indeed a good one, thanks for your effort! -- GrubbmGait Thanks so much for doing this. It looks great! --Corbos Nice spot on the NanoSatan problem, Grubbm. That variable was actually part of the targeting, so it was easily commented out. I was aware that it hit the walls a bit, and at the time I tested it I though that was OK, but now that does seem kinda stupid. I'm looking at RaikoMicro to replace NanoSatan. I should be done testing that within an hour or so, and if that works out, I'll post all the appropriate sub-pages here and move this chatter to /OldChat? or something. Thanks for the input, guys! I look forward to working on my gun again :) -- Voidious Alright, I actually went with DuelistMicro as the replacement for NanoSatan, as RaikoMicro was very similar to FloodMini and Cigaret, and even kinda similar to RandomMovementBot. I updated the table above, and it's looking good. I'm going to make all the TC sub-pages, and move this discussion once I'm done. I'll post the current results that I have, too. Let's rock! :) -- Voidious |
![[Home]](/images/RoboWiki.png)