#11  
Old 10-08-2017, 12:10 PM
carpboy carpboy is offline
Junior Member
 
Join Date: Jan 2017
Posts: 19
Default

Quote:
Originally Posted by contrary View Post
Are you looking at the story of the puzzle, or the puzzle ID itself? There can be many puzzles with the same story and same category values, which have different sets of clues. This website treats each unique set of clues as a separate puzzle, and keeps stats separately for each one.

I would expect the median times to settle out for each puzzle, and I would be surprised to see that much variation for a single puzzle. It would not surprise me to see that much variation on a puzzle story.
That's a good point, I was taking it to be for each 'story'. The little bit I have tried to correlate the same puzzle, on different attempts, was that it looked the same with only a slight adjustment at clues.
Reply With Quote
  #12  
Old 10-09-2017, 11:02 AM
uigrad's Avatar
uigrad uigrad is offline
Member
 
Join Date: Jan 2014
Posts: 99
Default

Yeah, if you pick a single puzzle size and a single puzzle difficulty, there are probably almost 1000 puzzles there, so getting the exact same puzzle again requires a lot of time playing.

There are 70-90 different "themes". I tried recording them all a while back, but no one seemed that interested in my results, so I gave up on it. It seems that there are at least 10 puzzles of each theme for each combination of difficulty and size.

If you play the 4x5 easy puzzles a lot, you've probably seen me record the medians in the comments. The medians are extremely important to me, because the score you get depends on just three variables, your time, the best time, and the median time. If the best time on a puzzle suddenly changes from 120 to 80 but the median remains mostly unchanged, then all the people who play that puzzle after that point will receive substantially fewer points. In general, we can expect it harder to get points now compared to the first couple months after new puzzles are introduced, simply because the top score changes dramatically over time, and the median hardly changes at all.

There are some individual puzzles that I have recorded the median 3 or 4 times for that puzzle, and I've noticed very little change. Over 3 months, it's rare for the median to change by more than about 10 seconds, but it does happen. I think the largest change in median after the puzzle had enough scores to show the median was about 40 seconds. You should play some of the 4x5 easy puzzles if you are interested in this.

As far as what median means, we all know the textbook definition, but I'm not convinced that this is what we actually get. One reason that I feel this is way is that I realize that to show updated median statistics, the database would need to keep track of EVERY single result. A lot of storage space could be saved by showing the mean instead. You would just keep the total amount of plays for that puzzle and the total amount of time spent on them. Of course the downside of using the mean is that a few outliers (like people who had 100 errors corrected on a single puzzle) would drive the mean to really bogus values.

Another reason I think the median might be bogus is case #1 in my image (back earlier in this discussion). If you notice there, the previous best time is almost the same as the median. In case #2, my new high score is actually slower than the median!! The only explanations I can see are: Case #1, all the people playing this puzzle just happened to have times that are clustered together, and case #2, there isn't enough recorded data for good stats, so purely fictional numbers are given. Even if these explanations are correct, I feel that there has to be some reason for the purely fictional numbers. If it was a round number and was always the same, then I would just assume that it is a guess from the admin about what the median should be. But, that fictional number does change, so there is clearly some hidden logic behind it.

My current assumption is that the median is always somewhat ficticious. The best time is kept (obviously), and a certain number of results are kept (possibly the last 10). If there are less than 10 results, then the database is pre-seeded with some ficticious times to give something to the first few people who play the puzzle. The median doesn't change very rapidly, so possibly it is moved by comparing how many of the last 10 plays were above the median and how many were below. If it is ever a 70-30 split, then it is bumped a second or two in one direction or the other.

If every result is saved in the database, then I would really like to see admin get rid of the fake bell-curve graph, and instead give us a box and whisker type of chart. In fact, I think the score should probably be based on your time vs. the 25% and 50% percentiles, instead of your time vs. the 0% and 50% percentiles.

Maybe I care way too much about this. Maybe I should just make my own puzzle site, haha.

Last edited by uigrad; 10-09-2017 at 11:19 AM.
Reply With Quote
  #13  
Old 10-09-2017, 03:33 PM
carpboy carpboy is offline
Junior Member
 
Join Date: Jan 2017
Posts: 19
Default

In my records, I have 90 (4x7, hardest level) that I have solved correctly, with no corrections. I know there are quite a few that I haven't done correctly yet.
Reply With Quote
  #14  
Old 10-15-2017, 08:47 AM
carpboy carpboy is offline
Junior Member
 
Join Date: Jan 2017
Posts: 19
Default

The only place I can find the puzzle ID is under recent games. However, in the games I've played since looking here, each ID is increasingly larger, I am wondering how these ID's work?
Reply With Quote
  #15  
Old 10-15-2017, 09:18 AM
fromalabama's Avatar
fromalabama fromalabama is offline
Junior Member
 
Join Date: Dec 2014
Posts: 4
Default

Quote:
Originally Posted by uigrad View Post
Yeah, if you pick a single puzzle size and a single puzzle difficulty, there are probably almost 1000 puzzles there, so getting the exact same puzzle again requires a lot of time playing.

There are 70-90 different "themes". I tried recording them all a while back, but no one seemed that interested in my results, so I gave up on it. It seems that there are at least 10 puzzles of each theme for each combination of difficulty and size.

If you play the 4x5 easy puzzles a lot, you've probably seen me record the medians in the comments. The medians are extremely important to me, because the score you get depends on just three variables, your time, the best time, and the median time. If the best time on a puzzle suddenly changes from 120 to 80 but the median remains mostly unchanged, then all the people who play that puzzle after that point will receive substantially fewer points. In general, we can expect it harder to get points now compared to the first couple months after new puzzles are introduced, simply because the top score changes dramatically over time, and the median hardly changes at all.

There are some individual puzzles that I have recorded the median 3 or 4 times for that puzzle, and I've noticed very little change. Over 3 months, it's rare for the median to change by more than about 10 seconds, but it does happen. I think the largest change in median after the puzzle had enough scores to show the median was about 40 seconds. You should play some of the 4x5 easy puzzles if you are interested in this.

As far as what median means, we all know the textbook definition, but I'm not convinced that this is what we actually get. One reason that I feel this is way is that I realize that to show updated median statistics, the database would need to keep track of EVERY single result. A lot of storage space could be saved by showing the mean instead. You would just keep the total amount of plays for that puzzle and the total amount of time spent on them. Of course the downside of using the mean is that a few outliers (like people who had 100 errors corrected on a single puzzle) would drive the mean to really bogus values.

Another reason I think the median might be bogus is case #1 in my image (back earlier in this discussion). If you notice there, the previous best time is almost the same as the median. In case #2, my new high score is actually slower than the median!! The only explanations I can see are: Case #1, all the people playing this puzzle just happened to have times that are clustered together, and case #2, there isn't enough recorded data for good stats, so purely fictional numbers are given. Even if these explanations are correct, I feel that there has to be some reason for the purely fictional numbers. If it was a round number and was always the same, then I would just assume that it is a guess from the admin about what the median should be. But, that fictional number does change, so there is clearly some hidden logic behind it.

My current assumption is that the median is always somewhat ficticious. The best time is kept (obviously), and a certain number of results are kept (possibly the last 10). If there are less than 10 results, then the database is pre-seeded with some ficticious times to give something to the first few people who play the puzzle. The median doesn't change very rapidly, so possibly it is moved by comparing how many of the last 10 plays were above the median and how many were below. If it is ever a 70-30 split, then it is bumped a second or two in one direction or the other.

If every result is saved in the database, then I would really like to see admin get rid of the fake bell-curve graph, and instead give us a box and whisker type of chart. In fact, I think the score should probably be based on your time vs. the 25% and 50% percentiles, instead of your time vs. the 0% and 50% percentiles.

Maybe I care way too much about this. Maybe I should just make my own puzzle site, haha.
Uigrad, I too doubt that they record every single score, which means that we're probably not getting a true median. I'm not sure how many scores they do keep, but I would expect more than ten. Maybe 64 or some other power of two. I think they could make a reasonable extrapolation by recording enough true scores to set the quartiles, and then just making a 1s adjustment up or down if a score falls outside of that range and ignoring any result that falls within it. Or they might use one or two standard deviations instead, and the 'nudge' might be more or less than one second, possibly related to the size of the puzzle. It wouldn't give you a true median but it might come reasonably close. I am quite certain at this point that they eventually discard old data. I used to have four unsolved 3x4 puzzles on my record from my internet connection crashing and I preferred dropping my solving ratio more than the penalty on average time. Recently, those records were expunged and it now says I have 100% solving rate for 3x4s. My guess is that they drop off after perhaps 4096 (2^12) records have been saved, because I'm nearing 5000 puzzles for that category and the offenders were from a long while ago. Maybe the number isn't based on powers of two, it could be 4000 games or something like that instead, but it does imply that they put a limit on the data stored for any given player and probably for any given puzzle as well, which supports the 'false' or 'approximate' median hypotheses.
Reply With Quote
  #16  
Old 10-16-2017, 07:39 AM
uigrad's Avatar
uigrad uigrad is offline
Member
 
Join Date: Jan 2014
Posts: 99
Default

Quote:
Originally Posted by carpboy View Post
The only place I can find the puzzle ID is under recent games. However, in the games I've played since looking here, each ID is increasingly larger, I am wondering how these ID's work?
The IDs that you see under recent games are useless. Every time you play a game, a new ID is played. If you get the same puzzle again, it will have a new ID.

The only way to know if you get a puzzle a second time is to leave a comment on it. At least, that's the only way I've discovered to know this. If you finish a puzzle, and see a comment left by yourself on a previous day, then you know you got the same puzzle twice.

People that don't leave comments only know when they get a puzzle a second time if the previous record was their own.

Quote:
Originally Posted by fromalabama
I used to have four unsolved 3x4 puzzles on my record from my internet connection crashing and I preferred dropping my solving ratio more than the penalty on average time. Recently, those records were expunged and it now says I have 100% solving rate for 3x4s.
Ah, very interesting! I would not have predicted that!

I haven't been watching my stats that closely recently. It's good to know that unfinished puzzles eventually go away, but that opens a whole lot of new questions. For example, is it only unfinished puzzles that are dropped by the system, or do stats for puzzles that you finished eventually go away?

There was one month (February 2016) where a whole lot of records were permanently lost. I had thought it was only puzzle records from that month that were lost, but for all we know, there may have been older data that was lost then also.

It would be nice if we got a little more information from admin about it. Everything here at this site works really well, but it just seems a bit abandoned.
Reply With Quote
  #17  
Old 10-16-2017, 10:36 AM
carpboy carpboy is offline
Junior Member
 
Join Date: Jan 2017
Posts: 19
Default

Thanks UI. I will try doing that. Do your personal comments remain when you as the user view the puzzle, or are they in a stack and get pushed out as more comments are made?

I would like to keep track of how I do, and since there are permutations on the same puzzle, of different difficulty levels, I would like to be able to ascertain when I was doing the 'same' puzzle again.
Reply With Quote
  #18  
Old 10-16-2017, 11:39 AM
uigrad's Avatar
uigrad uigrad is offline
Member
 
Join Date: Jan 2014
Posts: 99
Default

I wish there was a way to leave comments that only you would see in the future. But, the comments are more like a 1990's guestbook. You can write them and submit them, and they can never be changed or deleted, and they just pile up for ever and ever. For younger members who may not remember what a guestbook is like, haha, here's an example image:



You can see in that image that there are 220 messages in that guestbook, and you have to scroll through all the pages to get to the most recent. That describes the comments on these puzzles perfectly.

Fortunately, I've never seen 220 messages on a puzzle, they usually top out at about 15. As far as I know, they are never deleted. I've seen messages from early 2014, which is shortly after the new website was made.
Reply With Quote
  #19  
Old 10-19-2017, 05:17 AM
carpboy carpboy is offline
Junior Member
 
Join Date: Jan 2017
Posts: 19
Default

When the puzzle results come up, if you look at the page source in the area of posting a comment on the puzzle, there is a field called "QID". I wonder if that is the puzzle ID?
Reply With Quote
  #20  
Old 10-25-2017, 11:46 AM
uigrad's Avatar
uigrad uigrad is offline
Member
 
Join Date: Jan 2014
Posts: 99
Default

Quote:
Originally Posted by carpboy View Post
When the puzzle results come up, if you look at the page source in the area of posting a comment on the puzzle, there is a field called "QID". I wonder if that is the puzzle ID?
This is a great question! I hadn't looked through the HTML enough to notice this, so I'm glad you did.

Over the last week, I've been using Scrapbook to save the comment pages, so that I could look for any duplicates. I've played 4x5 easy puzzles exclusively during this time, and after saving 53 puzzle comment sections, I had no duplicate QIDs. My 54th puzzle was a duplicate! I checked the saved pages for the two that had the same QID, and those two pages were exactly the same, other than my score on them. The record was the same, and all of the comments on the page were exactly the same. So, yes, it seems the QID is unique for each puzzle.

For anyone who is curious on how I did this checking on Linux, I decided to write up exactly what I did. I assume this would probably work almost the same on Mac, or on Cygwin (for Windows).

The first thing I did was set up a quick-key for saving a webpage in Scrapbook. Figuring this out took some time. Go to Tools->Addons in Firefox, choose preferences for Scrapbook, and then choose the Keys tab. Now, as soon as I finish a puzzle, I can hit my quick-key combination to save the page. Note, once you leave a comment, it's too late to save the page, because the form containing the QID information is removed from the webpage to put up the "Thank you for leaving a comment" text. You can find the stored pages all in one directory. To see all the QIDs for pages you have saved, do this:
Code:
cd ~/.mozilla/firefox/*.default/Scrapbook/data
grep qid 201710*
Note here that the puzzles I saved were all in October 2017, so the filenames all started with 201710. You may need to change this if you do it during a different month. The output from the grep function is a bunch of lines that all look something like this:
Code:
20171025105216/index.html:<input name="qid" value="58673" type="hidden">
That is the 54th puzzle I saved. You can see the date is today (2017-10-25), and the qid value is 58673. So, the question is how did I know that I had 2 with the same QID? Checking this has been pretty easy. I made a .sh file and put two lines in there. The first tells me how many files I've saved have unique QIDs, and the second one tells me total number of unique QIDs I have. So, if I run the file and it outputs "42 42", then I know that I don't have any unique ones. When it says "53 54", then I knew that I had two that were identical. Here's what is in my .sh file:
Code:
grep qid 201710*/index.html | cut -d " " -f 3 | cut -d '"' -f 2 | sort -u | wc -l
grep qid 201710*/index.html | cut -d " " -f 3 | cut -d '"' -f 2 | sort | wc -l
It may not be obvious what cut does. -d " " tells cut to use a delimiter, and -f 3 says to return the third field. Looking at the line before, the output I get from the first cut function is value="58673". That's all good, but I want just the number, so that's why I have the second cut. It uses " as the delimiter, and takes the second field, namely 58673.

"sort -u" is simply a function that sorts the output line by line at only gives back unique lines. "wc -l" is the wordcount function, but I'm using it here to count how many lines I have (thus the -l). The second time I use sort, I leave out "-u" so that duplicates are kept. This is why I got "53 54" once I had my first duplicate. Once I knew I had a duplicate I could have used diff between the two outputs to find it, but since it was only 54 lines, I just decided to sort them numerically and look for it manually. The previous sort function was alphabetical, but I wanted it sorted numerically, so I used "sort -n" instead. It wasn't hard for me to find that 58673 was my duplicate.

Next, I found the files that had 58673 by using "grep 58673 201710*/index.html". Now that I had that, I used gvim to find them in the Scrapbook database. I typed "gvim ../scrapbook.rdf" to get open the scrapbook database, then searched for the 2 filenames that I knew were the duplicates. In there, I found the name that scrapbook had given those two entries, and I simply changed those names to "dup 1" and "dup 2" so that I could find them again easily.

It appeared that Scrapbook didn't like me editing that database file directly. It was going off what was in memory, not what was on disk. So, I used "F2-restart<enter>" in Firefox to restart it. After that, I pulled down my Scrapbook menu, and clear as day, I saw "dup 1" and "dup 2" in my menu. I loaded those pages, and verified they were the same!

Something else I noticed is that the QIDs seem to be ordered by when the puzzles were added to the database. For example, when I play a really old puzzle, it has only a 4 digit QID, but the newer ones all have 5 digits. Looking at all the QIDs I've seen, they lie very neatly in 6 ranges:
Code:
9107-9980 (900 puzzles?)
21062-21406 (500 puzzles?)
37009-37824 (900 puzzles?)
50005-50899 (900 puzzles?)
58100-58682 (600 puzzles?)
64790-64790? (100 puzzles?)
It's really weird that I only got one puzzle in the last range. I'm guessing that the last time new puzzles were added, it was just a very small number added in that batch (like 100), and maybe I just got one puzzle in that range. I don't really know. In any case, it seems as if there is approximately 3900 (plus or minus about 500) 4x5 easy puzzles in the database. If that's the case, you would expect to play approximately 62 or 63 before encountering your first duplicate (See Birthday Problem if you don't understand why), and of course that stat has very high variance (if you've had a stat class, you'll understand). For me to get my first duplicate at 54 puzzles, I ended up very close to the expected value. Isn't math fun?

Last edited by uigrad; 10-25-2017 at 11:49 AM.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 03:04 AM.


Powered by vBulletin® Version 3.7.2
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.

About Puzzle Baron

The Puzzle Baron family of web sites has served millions and millions of puzzle enthusiasts since its inception in 2006. From cryptograms to acrostics, logic puzzles to drop quotes, patchwords to wordtwist and even sudoku, we run the gamut in word puzzles, printable puzzles and logic games.