On some days it would work perfectly all day long, but on others it would stop working between those hours.
The biggest clue was it would always work perfectly on overcast days, but on sunny days this strange behaviour would manifest again.
Turns out the problem was related to the mouse being a cheap mouse. The case had very thin plastic.
The mouse was a ball mouse, and it worked by shining an LED into a sensor on each of the X and Y axes. On sunny days the sun would completely overpower the sensor due to the plastic case being very thin and on overcast days it would not. On sunny days the mouse would only work when the sun had moved around the sky to cast a shadow over where the mouse was being used.
Perfectly logical but baffling at first.
Turns out some versions of the Pangu jailbreak for iOS 7.1.x would crash during boot if the reading from the ambient light sensor was below some threshold. To this day I don't know the exact explanation of this bug, but it seems that Pangu included some unnecessary code that messed with the light sensor .
If you don't believe me, there is a huge reddit thread with a lot of people confirming this.
Called the cable company, tech came out. Everything inside was fine, but the cable from the main line to the house had a tiny cut in one spot, not enough to really affect the connection, but enough for ambient moisture to work its way in and foul the connection.
The other common denominator was the cable company refusing to believe it was an issue with their equipment; this meant it took a couple of months of calling them every night until they finally sent a technician and a manager to my house to verify that I wasn't wrong, leaving my house, coming back 15 minutes later to say "it'll be fixed tomorrow, there's a problem with the LE balance up the road" - and then the issue is resolved.
Now this doesn't sound so bad, until you learn that the first time this happened to me, i had only VoIP - so the internet would start to foul, i'd call the cable company, and the tier 1 would reset my modem at some point, and then i wouldn't be able to call back until after midnight (or whatever), when there was no longer a problem. So after a week of this, i would walk 30 minutes - one way - to a pay phone (remember those?) once the internet slowed, call them, explain that i couldn't do anything they wanted me to do physically, since they disconnected my phone line every time i called.
This is what happens with a de facto monopoly.
I will never pay suddenlink another dime, even if they're the only terrestrial provider, for whatever reason.
Almost every day, in the heat of summer, I get one to five 10-minute outages as soon as the temp gets over about 80F. More when it’s hotter, usually. Usually it results in a modem reset, so it’s hard to tell how long the actual outage is.
Been happening for going on 5 years. They replaced the under-street cable from our house to the junction box across the street to no effect. I suspect it’s that junction box, but afaict, none of my neighbors that share that junction box have the same issue. Not very fun to have your WFH day collapse unexpectedly in the middle of the afternoon.
Strangely, for the last month we’ve had several days of 80+ temps with no sign of outage. So fun.
Edit: yes of course multiple modem replacements and inside cable checks, to no avail.
I wonder if the coupler’s center conductor contacts could be expanding just enough to break the connection?
If you can, try forcing a level at/below the speed you get during the breakage and see if it just rides it out. If it does, shift it back up and plan your coffee breaks around it. Or don't, I'm not your mother
I've experienced something similar except for temperatures below ~32-36 degrees. At this particular location it would result in a ~1hr outage going below that temperature, but not when it went back above it for some reason.
Maybe there should always be a hidden option that only people that meet a certain troubleshooting ability threshold get access to when calling in for tech support….
I'm guessing that these scripts that we're all complaining about solve 95% of problems customers call in about. Sure makes things painful for the 5% of cases, though.
I've been a (grudging) Comcast customer for ~17 years, and I have been impressed by how their monitoring has improved over that time. It's been quite a number of years since I've had to convince them that I had an actual problem that their systems didn't automatically detect.
A customer's DSL connection dysfunction's frequency increased mornings and evenings. Culprit: the lift's electric motor leaking EM all over the place.
A bunch of DSL connections degrade when traffic increase... Crosstalk in big cables of course !
The sort of fun incidents that take a good while to troubleshoot... I'm glad we are migrating away from DSL to fiber: either it works or not !
They have all the data available on their end, as far as I can tell! (Unless DOCSIS modems somehow don't have a standard "signal receive report" functionality?)
Here's a 1960s vintage Automatic Electric line insulation test system at work in a step-by-step central ofice.  Here's the manual for automatic line insulation testing in a 5ESS switch. 5ESS is still the major AT&T switch for copper analog phone lines. After that, it's all packet switching.
For fiber, of course, moisture doesn't affect the signal.
This led to an urban legend: "bell tap". While Western Electric phones were designed to not react to the ALIT test signal, many cheap phones would emit some sound from the "ringer" when the 400V pulses came through, some time before dawn.
(I've sent a quick email suggesting it be added to https://news.ycombinator.com/highlights :)
If you have to build reliable distributed systems, it's worth understanding how this was done in the electromechanical era of telephony, where the component reliability was much worse than the system reliability. "Number 5 Crossbar" is worth reading, but hard to follow if you have no idea how telephone switching worked and are unfamiliar with the terminology.
Number 5 Crossbar, in current terms, was a collection of microservices. There was a big, dumb switch fabric, and "markers" which told it what to connect. Other microservices included trunks, originating registers (which listen to incoming dial digits), senders (which sent dial digits to the next switch), billing punches (which recorded toll call data for later billing), translators (which held routing tables), and trouble recorders (which logged errors.) Central offices had at least two of each resource, for redundancy. Resources were "seized" as needed from resource pools, with a hardware timeout and alarms to prevent resource lockup. If something went wrong in setting up a call, it was retried once, using different resources. If it failed on the second try, the caller got a fast busy and there was an alarm and a trouble recorder dropped a trouble card. Markers did not have persistent state. They started each call with a reset. So they could not get stuck in a bad state.
In the entire history of the Bell System, no electromechanical switching office was ever down for more than 30 minutes for any reason other than a natural disaster or a fire. It's worth understanding how they did that.
Maybe because you know that someone spent a lot of time on it before it was published since no adjustments could be made after the fact.
This feels like a term a sci-fi author would invent in an alternate history setting to replace "error log" and I find it very humorous.
The previous version was a panel of blinking lights called the "trouble indicator". When an alarm sounded, someone had to go to the panel and record by hand which lights were on. There were about 200 lights. So the trouble recorder, which recorded that info automatically, was added in larger central offices as an upgrade.
But every evening, mostly around 21:00 or so, the phone gives a gentle "peep" without then ringing.
I wonder if it's a line test?
There are various weird, obsolete signals in analog phones. Ring pulse alerting signal. ALIT test. Polarity reversal. Ring to ground. Ground start. Caller ID (1200 baud FSK between the first and second rings) DSL. Basic talk and ring was standardized around 1900, and everything else is backwards compatible. Ringers are supposed to ignore all that stuff. People who implement Asterisk PBXs are into this.
Here are some actual waveforms, if anybody cares.
And yes, it is an adding machine.
It might happen that an equipment manufacturer sees an opportunity and builds something, but then they have to go into a long sales cycles to convince operators to use it. Operators are in a duopoly situation in most places, so quality of service is kind of a secondary concern for them - customers may get annoyed, but as long as the competition is not vastly superior, few actually switch. It is not a market prone to innovation.
Tech showed up around noon, saw I was indeed having a bad connection, went and checked the signal at the junction box for the street (can't remember what you call these) and everything was normal there, so he closed it back up again and double checks the signal at the house again, but it was fine. He walks the lines to double check but everything looked normal.
His best guess was that moisture was condensing ever so slightly inside the junction box that morning, and was let out as soon as he opened it at around noon, which fixed the problem.
Eventually I called so many times and had so many appointments, that the tech lead gave me his direct number and told me to call him directly the next time it happened. When it did, I did, and he ran some tests, and confirmed there was a problem. I don't know that we ever got it sorted out (it was a while ago), but just getting them to agree there was an issue took a very long process.
There's an infrared beam and sensor. When the ice tray is full, it is supposed to block the beam, and then the machine stops making ice.
On a sunny day, there's enough bright light in our kitchen to fool the sensor so it keeps making ice.
We have a random magazine that we put on top of it to make it work correctly.
Same sort of problem. The obstruction sensor at the bottom of the door is confused by the strong sunlight and the door stops closing part way and re-opens.
I've tried a toilet-paper tube around the sensor but that isn't always successful. I really wish there was a laser sensor to replace it with.
Although, the possibility of a garage being oriented such that the sunlight would directly hit the sensors while the garage door is open seems like it could be a not infrequent occurrence.
It also is a lot easier to see fallen bolts and shit on a black floor than on a white/gray one.
You can, however, tape the sender and receiver together.
I had to replace my opener and door anyway and had a conversation with a tech about it. We decided on a LiftMaster in part because their sensors are very good at dealing with sunlight.
It looks like an industrial photoelectric sensor, including laser based ones run around $100, so maybe that can be a realistic swap.
But yeah, why this isn't laser based, or using a light frequency that is less affected by sunlight? Probably cost, or ignorance.
It'a not laser based so the sensors don't have to be perfectly aligned. Keeps your garage door working when you kid knocks it with their foot.
Also it could be 'fun' to swap out the LEDs?
One day we started having flaky tests, seemingly out of nowhere. We quickly identified that the issue affected tests involving graphical X client applications, but then we struggled to make further progress. The issue was just impossible to reproduce in other conditions... Well, as it happens, the CI jobs were running on some desktop machines we had installed somewhere within our premises. It turned out that some gentleman had plugged a mouse into one of the machines, and left it lying around on the shelf. Since then, when one of the machines was under a heavy load, the fans would spin faster, causing more vibrations, in turn causing the mouse to move, ever so slightly. And for ungodly reasons, this had side effects on tests.
Fun fact: the machines were not on my site, I managed to diagnose this over SSH. I was quite proud :-)
Let me guess - tests with very tight timings?
This makes me want to dig out the gitlab issue, and turn it into a better write-up! This'll have to wait until I'm back from holidays though.
It turns out our solar panels (or the optimizers, or the inverter) emit radio frequencies that interfere with our garage door opener. When the sun is out and they are producing energy, the interference is stronger than the homelink garage door opener.
A few years ago the garage door openers started working fine. It took a few days to realize it was because the inverter had failed.
I’m fairly certain there are some FCC regulations that would require our installer to fix it, but that relationship soured during installation and I’d rather deal with an unusable garage remote than dealing with them for warranty work.
If you have any amateur radio neighbours they'd probably love to help you with a project like this.
I have a HomeKit opener attached to it that we use during the day. Fortunately that’s been reliable enough to get around the issue.
- Use to fix PCs professionally in the early 90s.
- Guy comes in with PC. Right-mouse button stopped working.
- Replace mouse. Still not working.
- Play with Windows 3.1 drivers. Nothing helps.
- Pull HDD from another PC, install, boot. Mouse button still broken. WTF.
- Pull whole mobo, put another spare mobo in, with replacement HDD and replacement mouse. Still don't work.
- Replace PSU. Right-button works.
- Give up on computers, live in wilderness, eat squirrels.
The first was a VDSL connection I had at home. It worked great (fast, for the time) except when it didn't. It always failed in the evening. Techs would come out, bless it as being good, and leave -- because of course it worked while they were there. Unless they showed up and it was broken and then they'd declare that it was an outside problem, and that they'd have to get someone else to fix it (because the residential techs can't do overhead work).
I made lots of (very polite) phone calls, which results in more refunds and more service calls. More than once, my driveway and the street in front of my house looked like an AT&T convention.
This went on for months.
I had direct numbers and emails for tier 3 support and the local manager who oversaw this plant. We were all getting to know eachother too well, and there were boots on the ground addressing this problem as many as three times in week.
I eventually noticed that as the days got shorter so did the evening outages...and that if it was a cloudy day, then that day was often outage-free.
I had an epiphany: The problem might correlate with the angle of the sun, and the duration of exposure!
I checked my logs and the past weather, and sure enough: It lined up.
So I reported my findings, even though they seemed like nonsense as the words came out of my mouth, and they sent out some crazy-haired guy with bluejeans and an untucked shirt who was clearly not used to wearing a uniform, and who was also obviously not normally customer-facing.
"I know exactly why they can't find the problem," he said after I reiterated what I'd learned. "Your neighborhood still has old lead-sheathed overhead lines, and nobody knows how to work on that anymore."
"But I'm certified on that. I'm going to go back to the shop, pick up a bucket truck and get your line fixed. It will take me most of a day to do this, but I will be back when I'm done."
And it was getting pretty late, but he did come back to let me know that he found some things and fixed them. And I don't know what those things were, but it was fine after that -- and it stayed fine.
Thermal expansion letting cosmic rays leak into copper pairs wrapped in paper, tar, and lead? Who knows. I certainly don't know.
I've never encountered that stuff professionally (and it isn't your grandfather's 25-pair cable) and as this dude said, "nobody knows how to work on that anymore."
On some days, exclusively in the morning hours, the printer would fail to detect the start of a new label, printing over several labels.
After connecting remotely and checking the usual (queue, network connection, drivers etc), I asked my colleague to call me, as soon as it happened again.
When I went there, I saw that a ray of sunlight hit the printer. The windows had shutters, but there was a gap.
Label printers detect the gap between labels using a laser. And for some reason, the printer's case had a clear window at the top.
I printed an empty label and stuck it on the little window.
I would have guessed they'd shrug at the first sign of trouble, swap it out with a known-working mouse and mark the ticket resolved... unless all the replacement mice were thin plastic too, I suppose.
It would bother me until I figured something out.
I suppose a good office-computer mouse in 1990 would cost $100 ~ $200 (say $350 today). In that case, yes troubleshooting it for a day would make sense, especially if it's not an isolated problem.
This is not a sensible comparison to make. Support staff have a lot of free time. They have to, because they're support staff -- if they were always busy, then whenever a problem arose, it would be impossible to get support.
So to have the IT department playing with the mouse is unlikely to cost the company anything. If something comes in that's more important, the mouse problem will be put aside. If they have nothing better to do, they can play with the mouse.
Seriously, this is one of those dumbass stories that come from your boomer relatives with the subject line "fwd: fwd: fwd: fwd: fwd: fwd: fwd: fwd: re: fwd: fwd: fwd: fwd: fwd: fwd: vanilla ice cream"
Nobody actually believes this story is true, right?
In fact, I'd argue the quaint style of the story does geeks a favor: if it's appealing to normies, maybe they'll appreciate us technical folks' perspective a little more.
lots of people, both technical professionals, and non-engineers who are observant and have an appropriate level of belief in causality, troubleshoot transient failures like this all the time. a difference in the amount of time between shutting down the engine and starting it up is one of the first things that someone like this would test, or control for. it's beyond implausible that the second time the guy got vanilla (after riding along for 4 trips, two long and two short), the engineer didn't raise the question of how long he was in the store.
people troubleshoot things like this by being able to separate causes which are plausible, although unlikely and surprising, from things which aren't remotely plausible. the 500-mile email story and the stories above about sunlight interfering with sensors demonstrate this.
if you're the sort of person who believes that the type of ice cream you get might affecting your car's ignition - the type of person who buys ice cream often but never thinks about how long the errand takes, you simply never get the point of being able to make a pattern between those two things. the second time your car doesn't start, you blame it on the scratch-off lottery ticket you won $2 on which used up your supply of luck for the day. the third time, you conclude that the car ignition knew you were late and likes to choose its failures to cause maximum annoyance. and the fourth time, you realize that your mother-in-law gave your car the evil eye that morning.
the story as told, especially when presented as a real parable about engineering rather than an amusing myth, is frankly insulting to the other type of person. the untrained, not necessarily educated person who cares about machines and believes in material reality. the person who starts checking their watch each time they go to the store and a couple of weeks later is telling their mechanic friend "if it's more than 3 minutes or so, it's fine. but if you try and start it before 2 minutes, then you have to wait another 5 before it's ready to go".
And the engineer is sitting in the car on the first night and it "wouldn't start," which signals the end of the episode for the day. Were they stranded? Did the car start after a few tries, which would have given a huge hint about the root cause? Surely the engineer who had reproduced the issue would quickly narrow it down by running diagnoses on the car itself.
But the family dynamics are the most improbable part here. How does the family have this predictable routine and not simply stock up on ice cream? The family has enough kids that the consume a whole $unit of ice cream per day. So with that much chaos in the house, how does the dad justify going out for a drive after dinner when the chaos of family multi-tasking (cleanup, chores, homework, bedtime) is at its peak? "Oh, look. Out of ice cream again. I'll be back in a few!"
The manufacturer is unlikely to see a problem with putting a display of very popular products right at the front of the store where people can't help but see it.
Or perhaps, unbeknownst to everyone in the world who uses the word 'vanilla' to mean 'mundane', vanilla ice cream actually enjoys far higher profit margins than all other ice cream.
It would be literally impossible for an aisle cap to feature every ice cream flavor available - there are so many that each flavor would have very little representation in the display, and the concept would fall apart as soon as anyone bought something from it. At that point, you're paying a bunch of extra money to send the message "check out our least popular flavors".
What's not implausible is the idea that one flavor of a product with several flavors might be located far away from all the other flavors. That happens all the time.
That's what I expect them to do, though. The most popular flavors, by definition, needs advertising the least.
The advertising costs the same whether you advertise popular flavors or unpopular ones, but you'll get a lot more sales by advertising the popular ones.
Go check out a grocery store, see whether the aisle caps slant towards popular or unpopular product varieties.
They still work perfectly... except for a regular pop of noise every few seconds that would intermittently show up, that scaled with the volume setting.
It turned out, their portable phone (read: landline with short-distance wireless RF handset) would ping from the base station to the handset, if it were off the cradle, which was being picked up by the unshielded line-level audio cable and amplified.
Moved the base station further from the cable, pop disappeared.
But this is different than the old <2g/edge phones, which wouldn't interfere unless you were about to get a call - because the tower said "where's this phone?" and your phone would max out it's tx and say "here i am!" and that's what you heard. This is probably incorrect, but based on my observations this is what occurred.
Remember the doodads you could put on your startac style phones on the antenna bit, with LEDs in them - they'd light up when you were about to get a call, as well, by design!
And Mom ran on coffee. Lotsa coffee. Sometimes I think she just did it because she didn't hear enough complaints coming out of the speakers.
I rejoiced the day that thing died. She's now got an even beefier one, but it doesn't interfere on the 5ghz bands at all, and I'm not testing the 2.4 out of respect to the spirit of that ole menace.
We were developing a smart camera product which were counting traffic on a road. So for example a city council would install this camera somewhere on a road and it would generate statistics of how many lorries, and passenger vehicles, and motorbikes used that road.
One of our cameras exhibited a problem where it restarted every day around roughly the same time. It wasn't exactly the same second though, in fact there was a clear pattern to it. One day it would restart at 19:12:10 and the next day two second later, then again the third day two more seconds later. (not the real timestamp and i don't remember the real time deltas either, but there was a clear progression)
After much debuging we learned that the issue was that as the sun was settling some street furniture projected a shadow in front of our camera. Our software wrongly concluded that it is a vehicle and started collecting information about it for classification. But of course shadows creep a lot slower than real vehicles so it run out of memory before the "shaddow vehicle" has passed out of the frame. And once we run out of memory the system froze and then got restarted by a watchdog.
Turns out the pattern we have seen in the timestamps was caused by the angle of the sun changing which made the shadow trick our algorithm just a little bit later every day.
1. http://www.st-v-sw.net/Obsidian/Martin/gravity.htm (search sprinkler)
The ice cream story reads too much like a dramatisation to be truly believable, but accidental and repeated unplugging is common I expect.
The UPS on a server I manage would trip once a week around the same time. The old story came to mind and sure enough, once a week it was time to vacuum and someone would plug in a vacuum cleaner into the same circuit (700W vacuum cleaner on a 100V/15A circuit), causing enough voltage dip to kick the UPS into gear.
But, the key thing is that urban legends always reflect some sort of underlying societal belief or worry. "Janitor polishes off patients" is about the fear of death, and how it can come for anyone at any time, and there's nothing they can do about it. "Janitor unplugs server every night at 5" is probably more about the idea that strange things happen sometimes, but it usually turns out that there's a good explanation.
Sorry but even before today, having a automotive engineer sent to a random person's home over what clearly sounds like a quack letter seems implausible to me. The dictates of capitalism, human resources, and the politics of the workplace would make this difficult if not impossible. Even in the past when there was more human capital in support positions and more of a sense of customer service.
Way, way too many suspicious stories involve high-level people being involved in trivial issues. I just find it all pretty suspicious. Real stories tend to start with poor customer service at the dealership and being mocked by managers and mechanics. Not some unrealistic ideal white knight manager sending off engineers to people's homes. Imagine how many weird letters a place like pontiac gets. They don't have the manpower to do this if they actually chose to do it, and engineers might balk at the idea of doing at-home support too.
Pretty much any "idealized Americana" business story should set off BS alarms in us. "Oh a trivial problem with your car? No problem ma'am, I'm sending our top engineers over tomorrow," doesn't happen because its costly and unsustainable. Instead ask anyone who has odd car problems. Its endless painful calls and visits to dealerships and mechanics. There's a reason we have lemon laws for cars. Its because whats described in this story doesn't actually happen and people demand restitution.
I don't doubt that someone had a famous vapor lock shopping story (ive heard different versions of this story, usually about a housewife picking up her child from a nearby elementary school), but over the years these stories get modified into memetic structures based on dishonesty because most people are social capital seeking and having a humorous story provides them the immature ego boost they need. So "wow my car had vapor lock when I make quick trips" became "So the CEO of Ford came to my house to look at my ice cream car..." The latter is just more interesting in the market of storytelling.
That is to say, the ONLY reason this story is here is because its been modified to be memeticly attractive. A "boring" (i find old technology faults interesting, personally), but a "boring" story about vapor lock wouldn't make it to places like HN or reddit, which are memetic responders (upvote/downvote mechanisms) and lowest-common denominator (by this demographic) popularity machines. But dress up that boring story and now everyone is repeating it, often times claiming its their story and they know the people in it! The same way the comment you're responding to probably doesn't actually know the famous "unplug the server at 5pm" person.
I remember one incident where that "fix it" letter came from high enough that I drove out to the customer's house and swapped out their radio in their driveway. At night.
When GM bought EDS, Ross Perot ended up on the board. He'd do all sorts of silly things. Like when something went wrong with his car, he'd take it to a dealership. And report back to the board how that went. The first few times, they just told him to hand the keys to the valet at the executive garage and tell them what needs fixing. The plant I worked out of made radios. Other branches of the division that I worked for made engine computers and instrument clusters. If someone at the executive garage had a radio problem, one of my tasks was to go out to the assembly line, grab a radio, test it, then get it to the Kokomo airport for the GM jet to pick up. FedEx (tagline: when it positively has to be there the next day) wasn't fast enough. That fleet of jets would carry parts from plant to plant. And the executive garage was the best equipped and best staffed GM dealership on Earth.
So, in essence, "it depends". Good stories, in the memetic sense, will have hooks to ensure that the moral or point of the story is remembered; in a great story, the memetic hooks are so great that you can repeat the story nearly verbatim to other people, after hearing it yourself.
We were only selling equipment in the US at the time but some had found its way around the globe. This one particular Indian gentleman had been engaged for some time with support claiming that his unit wasn't working right and exhibiting all kinds of strange behavior.
I had a habit of looking at the cases with the longest open history, which is how I found it. I continued to monitor though I didn't send anyone out to India or anything like that.
Eventually it came out that the sleek looking aluminum unit had some kind of stocky packaging material stuck to it, so the customer had put it in the washer to clean it off.
Comedians make up stories all the time to entertain audiences. These stories don't require accuracy, they are more about delivering specific results; a laugh, a story to share, confirmation bias, etc.
Many people lie, believing they're telling the truth. I think you will have a hard time truth policing people who don't and won't care, but focusing on truth and validity is probably a useful skill for you in many parts of your life.
There is a lot of gradation, the differences between stating misinformation you believe is true vs stating what the majority believes to be true due to laziness vs an intentional lie, etc.
And so it doesn't become apocryphal but stays a real story, there 100% were people who "Fixed" their RRoD Xbox 360s by wrapping them in towels for an hour so they cooked themselves even more, or they put the mobo in the oven on a low temperature. That fix was also 100% used for fixing certain graphics cards unseating in the mid 2000s as well.
One of the things I talk about is that these stories often ride the edge of plausibility. Even with this one, dispatching an engineer to someone's house over a "quack letter" seems like a thing that could happen or could have happened in the past. Having high level people involved is a thing that others have talked about, but I'd also like to point out that people have literally emailed firstname.lastname@example.org before about issues and gotten resolutions that way, so it's not completely off the mark.
The real clue here that we're dealing with an urban legend and not an actual incident (though it may be based on one) is the lack of specifics. Urban legends are often not really pinned down to any specific time or place.
The vapor lock ice cream story might go back to this 1997 Car Talk Puzzler, where Ray claims it was a customer of his: https://cartalk.com/radio/puzzler/finicky-volare
Do not mistake cynicism for intelligence. It is actually negatively correlated [0, + other studies]
Just listing a basket of "I doubt..." and "No one does..." does not make it all false.
Of course Just So stories exist, but that does not mean that all cool stories are fabricated Just So stories. In this case, it is the cynicism that merits a [Citation Needed] tag.
You're right -- it was obviously a fun engineering legend, and leave it at that.
It turned out that the lawyer next door would come in, turn on his PC, printer and coffee pot simultaneously because they were all on the same power strip, and the drain was causing an undervoltage on the circuit the server was on during startup. We had it on a UPS, but it turns out that at the time consumer grade UPS systems only handled outages.
I measured drops as low as to 85 volts, in practice anything under 95 or sou would reboot.
Back in the mid 90s, I built out a system that gave every school in a district their own webpage that was carved out of some government funding for providing internet access. There was no budget for hardware though, so it ended up running on a repurposed workstation in somebody's office. One Tuesday even the cleaners unplugged it to vacuum and it didn't power back up after being plugged in. On Wednesday somebody helpfully stuck a piece of paper saying "don't unplug" to it, which seemed to solve that problem until the whole project was mothballed.
In the late 90s, I worked at a company where we started getting complaints from the staff about machines being getting slower over time. Nobody took it seriously until there was an inventory of machines taken and we found that a large amount had significantly less memory installed than they should have, somebody was stealing half the memory sticks from each. Hidden cameras were installed in the office and it turned out that somebody on the cleaning crew came with a screwdriver and ESD bags and knew how much to take to leave the machines working.
Had he struck once or twice and then left the rest alone nobody may ever have figured it out.
less than an hour later, we get an email from nagios (i think it was nagios? it was a GOOD while back) complaining the server was offline. we got into a cab and went back straight up (this server was not supposed to be offline ever).
guess what? a maintenance guy turned the server off by mistake while cleaning up the server room -- even worse, he was not even supposed to be there!
this triggered a bunch of security checks and the company found out that most employees had access to any room in the building.
The hardware consisted of 6 color terminals (Wyse 350s I think), the server, and 10 PCs with large, 20” tube monitors. We had these all in my bosses office powered through assorted outlets and power strips. The room was wall to wall machinery.
It was after hours when the janitor came in and plugged in his vacuum. I looked at him, he looked at me, I looked at my boss, who looked back. We then both turned back to the janitor, who looked at us.
Not a word was said as he fired it up. 2 seconds later, the power was out, we blew the breaker. Took us a half hour to get it back on. But it was funny at the time.
Huh? Yesterday? Or are you referring to the fact that servers are now often offsite, in 'clouds'?
hispital ones are red wires
Brother-in-law was known to be "not so good" with cars - so his automotive engineer father didn't take the complaints seriously.
Complaints and emotions escalated, until brother-in-law convinced his dad to swap vehicles for a month, so (he hoped) his dad could experience the problem for himself.
After the problem manifested in the parking lot of the GM Technical Center, and the whole crowd of GM engineers surrounding the vehicle couldn't figure out why the heck the electrical system seemed to be dead, my brother-in-law felt pretty vindicated.
Had it towed to a service center a few times when this happened. Every time, by the time they got around to trying it (a few hours later)... it would start fine, with nothing to diagnose.
Then, it was 99% of the time it was fine. I was with a group of folks car-camping off road to fly a human powered airplane for a couple days, and... no start. Finally started -- with no sign of any problems -- around noon the next day, and I high-tailed it out of there a day before the rest of the group, because getting stuck there /after/ the rest of the group would have started pushing my comfort level. So at this point it's actually interfering with my life.
I've tried all the usual stochastic troubleshooting (swapping out fuses, light to moderate percussive maintenance, alternate keys) and nothing. Finally it fails to start in my driveway, and I get it towed to an independent mechanic. It's short tow, and it fails to start when it gets there! So now he's seen the problem, and is as puzzled as I am. Of course, when he tries again the next morning, it starts fine.
He proposes two possible fixes: replacing some ECU module, or replacing the fuse box itself (under the theory that it's the connector or connection into the bottom of the fuse box that is having some moisture ingress or intermittent connection). Of course whatever we choose, I won't know if it was right or not until the next time I'm stuck. The ECU is multiple thousands of dollars, and the fuse box is < $200 with labor, so I make the easy choice.
This was six or seven years ago, and that car is still my main and only car. Hasn't had a single mechanical issue since swapping out that fuse box. A good independent mechanic and a good guess!
Finally figured it out after months when I moved a wire and it started, then move the wire close to another wire and it would no longer start. These were wires that would typically be close, so my guess is one of the wires was now generating enough noise that it was bleeding over into some other system and causing it to fail. As the car bounced around the proximity of the wires changed and lead to the random behavior.
Turns out contactors pulling in and out a 5000w load generates some strong EMF and sometimes that EMF is enough to cause random glitches to the CPU or other hardware.
Switching to high power solid state relays completely solved the problem while keeping the system compact. The actual silicon transistor was so big you could have drawn the mask by hand and it was attached to a heat sink half the size of an adult fist. I was initially worried about reliability but (knock on wood) 8 years later the system is still working without issue.
One of those things where if it had happened 100% of the time we'd have figured it out quickly. But it was so infrequent that no one thought of that as a cause.
We had one access point (unifi) in our datacenter which was consistently failing.
fail, RMA, fail, RMA, probably replaced that thing 5 times. It was also incredibly unreliable.
Meanwhile all the other access points (probably 10 of them in total) had 0 issues.
Eventually realized that the cable for that AP was running perpendicularly over the conduits which fed power into our suite, so, about 1MW of power. Relocated the cable so it was farther from the conduit and it never had an issue again. Makes you wonder about the effect of working in such environments.
I know an older guy at church, whose kids all graduated college - except for one.
His "failed" son is the top mechanic at a Mercedes dealership. He does some supervision, training, etc. But the reason the dealership is paying him $200K/year is his skill at figuring out and fixing problems like that, for the dealership's most desirable and profitable customers.
(That I've heard, none of the mechanic's "successful" siblings are making that kind of money.)
I know a story of a certain large engineering firm, dating back to the second World War. They had a senior engineer who habitually came to work drunk and slept through meetings. Every once in a while, they'd wake him up, and he'd save them a couple of million dollars. He had a gift for finding clever solutions.
He probably would have had a better life if he'd gotten his act together. But knowing how to fix subtle issues, or how to design good processes, can be a ridiculously valuable skill.
The easiest diagnosis is to rotate the cables on the terminal several times to rub off oxide build up, then leave them in an orientation so that the natural tension of the cable forces the clamp into good contact.
A "dead" (low voltage) battery will still cause some indicators/lights to come on when you crank, while a bad connection usually acts like zero volts.
The car was dead for long enough that I could get it towed to the mechanic and they educed it was a problem with the EIS (electronic ignition system).
The EIS computer was sent off to mercedes for diagnosis, they reported that the computer itself was fine but it was an issue with power. The mechanics traced it back to a bad connection of a wire somewhere.
Any of the other positions it works just fine.
Once he got to Missouri he noticed the transfer case (I think) had somehow bounced out of neutral and was partially engaged, which chewed many teeth off the flywheel. To start it we had to get a pipe wrench and manually rotate the flywheel to a spot with enough teeth to catch the starter and turn over. That got harder and harder (as it kept chewing off more teeth) so eventually we had to push start it and pop the clutch. Thankfully it was a manual so it still worked!
I don't know if Alec Proudfoot counts as a normy, but DaSH PA flew a bunch of times, with many different pilots. That particular event, which was an attempt to try some records, was a bust; there had been rain recently and the dry lake beds weren't. Landing gear was a recurring problem (light weight and robust don't go together), and even some last-minute attempts to build a runway out of 4x8 plywood sheets was unsuccessful.
I have some very, very fond memories of flight days at Moffett, though. I had gotten to work on the Moffett runway for work previously (so cool out at the bay end of the runway at night, totally silent), and getting back there to help run ground ops (including assembling DaSH PA before sunrise so first flights would hit the calmest possible air) was just lovely.
That GM vehicles from this infamous era would suffer from maddening, mysterious electrical glitches makes perfect sense.
A formative early experience of mine was learning valgrind to track down a Heisenbug for a C project I was working on (which turned out to be an invalid read in a dependency). I’m indeed thinking of this anecdote when generalizing about whole-systems failings, since troubleshooting memory errors is so difficult.
I think there’s an analogy to be drawn when designing large systems on top of unsound foundations.
The higher-level point is that Heisenbugs are an emergent phenomenon of complex systems when fundamentals are lacking.
* C systems are lacking because the language is very old and we’ve learned that we need additional infrastructure to avoid memory errors.
* 1980s GM systems were lacking because of a management culture which didn’t value reliability, leading to inevitable issues in e.g. poor grounding and electrical isolation.
It’s my belief that many contemporary tech companies have management cultures similar to 1980s GM, and subsequently waste tremendous resources when troubleshooting complex systems which are not designed to facilitate troubleshooting. That’s why the original article resonates strongly with me.
Mechanically they may be reliable, but 90’s GM forgot how to make paint stick to metal and had to pay to repaint a massive number of vehicles that simply pealed if parked outside for too long. How?
And there is absolutely no forgiveness in my soul for the Chevy Citation. I joked when I moved to Seattle that the main problem is since there is no salt, there are still Citations on the road and that is unnatural. Their place in the natural order is the junk yard.
The first call to GM revealed that it was the acid rain (acid rain in the 1980s was what global warming is today -- the cause of all evil). Exposing my car to rain voided the warranty.
The second call to GM revealed that ultraviolet light destroyed the bond between the paint and the primer. Exposing my car to sunlight voided the warranty.
I spent hours researching this issue through the trade mags and published court filings. Plenty of legal findings about implied warranties of fitness for purpose. Evidently GM had an unpublished policy that it would pay the cost of a repaint to dealers for this situation, and the dealer was expected to provide the work for free.
Of course, the greasy grin of the dealer as he quoted full price while knowing he would collect the same from GM was enough to make me drive the car with no paint for the next 10 years so everyone could see, and recommend nobody purchase any GM product ever again.
I'll say this though: that primer sure prevented rust.
But based on what peeled, I’d say thermal expansion or UV damage were involved. The former could still be the primer’s “fault”.
This was the fate of many British Leyland cars, even the ones that people genuinely liked such as as the classic Mini and the MGB were practically hygroscopic.
BMW is hyper aggressive about the brand, including going after each and every use of the brand for the original cars, even when used in the context of spare parts.
Of course there are: "a mention of a praiseworthy act or achievement in an official report, especially that of a member of the armed forces in wartime" Don't focus on the North American usage of "a traffic citation". Citation is almost a contranym, which is a word that has at least two meanings that are opposites of each other, i.e. bolt, bound, buckle, cleave, clip, consult, ...
Those are named after a race horse. Car may also be for the race horse. Or maybe the car’s named after the plane(s).
Sure, the name is derived from "kobold", but that's like saying you should never call anything good "terrific", because it derives from the root "terror". Etymology isn't destiny.
> According to the story, when Chevrolet tried to market the Nova in Spanish-speaking countries, the car reportedly did not sell well because in Spanish, "no va" translates to "doesn't go". This led people to joke that a car named "doesn't go" wouldn’t be a popular choice.
> However, it's important to note that this is largely a myth. In reality, the Chevrolet Nova was relatively successful in Spanish-speaking markets. "Nova" as a word is understood to mean "new star" in Spanish, and it's unlikely Spanish speakers would naturally break up the term into "no" and "va", just like English speakers wouldn't naturally break up "notable" into "no" and "table".
> But the story remains popular as a cautionary tale of the consequences of not considering linguistic and cultural differences when naming products for international markets.
I'd guess both the car and the plane owe their names to the horse.
Relatively famous business jet.
A few other GM vehicles have this issue, Chevy in particular. A well known example is the market failure of calling a car Nova (No-Va) in South America.
> Are there any good connotations of that word?
You should know what they say about the Mitsubishi Pajero, too!
Doesn't seem to affect sales, though, probably like the Chevy Nova. Those Kia are everywhere.
Could be an older component stopped being available. Like when Apple switched to environmentally friendly lead free solder, but then the NVidia laptop GPUs got so hot they unsoldered themselves.
Dodge Ram Trucks had exactly the same problem. I also have no idea how two different companies with 100 years of history simply forgot how to paint cars.
American carmakers really needed that kick in the ass from Japan. Around 1990 was when my parents went from being protectionist, "buy American" to never buying another domestic car again in their lives. They were angry, angry at the reliability difference and angry knowing that domestic carmakers could have done better but instead relied on people like them to buy the flag.
Granted the traction control and anti-lock brakes occasionally fail to boot up, but it seems to be a pretty good car, but it was close to the top of the line. Gets 27 mpg which is not bad for a big ass car. I like how it has a lot of the feel of a 1970s boat but it has airbags, OBD II and most of the good features of a modern car... And we didn't need to get a loan to afford it. Driving home though I was looking in the mirror and seeing it dwarfed by today's XXL trucks and SUVs.
Like turning on the backup lights in a parking lot when the engine isn't even running.
On Consumer Reports' list of car brands by reliability, none of GM's brands even crack the top 10. GMC and Chevy are 20 and 21, respectively, out of 25 brands. (The top 5 include, unsurprisingly, Toyota, Lexus, and Honda—your classic reliable Japanese brands.)
 https://www.consumerreports.org/cars/car-reliability-owner-s... (may be paywalled...?)
It's been long enough that I don't remember what I did to get it shut off (maybe I toggled a circuit breaker?), but when I got back to base, I made a note in the airplane log about the problem and also left a note for my instructor, who was out that day.
When I came in for my next lesson, instructor mentioned that the next person to use the airplane, also a student on a solo cross-country, got stranded 100 miles away because when the engine wouldn't shut off, he panicked and pulled the throttle hard enough to rip the cable through the firewall. Airplane had to be put on a flatbed to get it back home.
Guess he didn't see my note!
Usually there was a fuel disconnect along with the throttle that would eventually starve the engine.
I just tapped the relay with a wrench and it un-stuck and turned off.
Funny enough that was almost 5 years ago, and it hasn't done it once since in more than 80,000 miles of driving. Not in -35C, not in +45C. Odd little relay.
My (GM) car gets really funky startup behavior when the battery gets old. It will often turn the starter fine, but the electronics can get stuck in weird states until I disconnect the battery (essentially a hard reset).
The dealers wanted a ton of money to diagnose the issue, even though I suspected it must have been the infotainment system draining the battery. I just ended up replacing the infotainment system with a cheap CarPlay compatible one and the problem went away.
Problem is I cannot get money from the class action settlement since the original infotainment system is already out and I fixed it myself.
You only fixed it because it was a problem though. Not a/your lawyer though, so good luck and have fun navigating the American legal system!
What I do not understand is how the settlement did not require Subaru to compensate all Subaru owners for the faulty design. Why was there proof required, other than the purchase of their car with the faulty components?
I’m sure we’ve all been caught trying to troubleshoot a problem where the actual issue was a loose cable.
Lots of programmers that would struggle to diagnose basic stuff when their laptop when it goes wrong.
In that house,on certain days of the week at midnight, something that sounded like recordings of political speeches from WW2 would be audibly (but faintly so) from all of stationary electronics. It was so faint, that it was usually hard to pinpoint any one source, and the words used weren't understandable, so it usually sounded like it was coming from everywhere all at once, and like it was an old recording.
This would happen even if they were unplugged. Even if my power went out.
I though I was either going crazy, or my house was literally haunted by hitlers ghost.
Well one day I was in my car and recognized something on the radio that reminded me of this spooky problem I had.
It was the signon of a Catholic am radio station that opened up with Gregorian chanting, and a sermon. This signon happened at the same time on the same days of every week.
Turns out, the wiring in that house was somehow functioning as an am radio receiver, and some common components would vibrate out the audio encoded in the radio signal.
A pair of old not-plugged-in computer speakers can also be an AM radio receiver.
Every time I played on his laptop, this did not happen. He swore he was cursed.
This went on for many days, with many instances where it would happen for him but not for me. Then one day I just sat and observed him while he played, looking for any difference. That's when I noticed his watch band is metal with a magnetic clasp. The position of his wrist on the laptop was tripping the hall sensor, making the laptop think the lid was closed.
Him and I (and his mother) were glad to find out he is not cursed. :-D
word to the wise - never use the shutoff switch on anything with a carburetor - use the fuel shutoff valve. This prevents gasoline from varnishing your carburetor bowl.
I probably won't use that word again for a year, now, heavens.
So, that sounds like a carb rebuild, right? Come to find out that for about $17 shipped a seller on eBay with something like 200K feedback will sell me a brand new carb. Fitted that, it starts on first pull. There's absolutely no reason to mess around with a rebuild on something like that.
I will always clean/rebuild carburetors before replacing. And usually when replacing I will explain to the customer and get permission to spend their money on an OEM one.
Looks like the OEM part is only $50-some, so worst case I'll get one of those down the line, but the generator was running at a friend's house for three days straight a few weeks back and no issue, so hopefully it'll keep being good.
I'm absolutely a car guy, and I'm 41 years old.
Strangely enough, I've never owned a carburettored engine, and it seems unlikely I ever will (except, maybe, for a chainsaw)
I think it was a Renault 11 and I'm not exactly sure how old I was, probably a bit before I got my license, so late 90s.
To me the moral of the story (and my experience) is: user's problems are usually real, but don't trust their ability to diagnose the actual cause.
I have the role of "chief troubleshooter" in my engineering group. I have a bit of a nervous tic that forms when I hear absolutes like this.
Don't assume the customer is always an idiot. Don't assume ANYONE is always an idiot. It limits you as an engineer.
Listen to everything. You don't have any need to abide any of it, but listen to it all.
I've had some of the least-qualified people throw out something which absolutely ended up being insightful, although sometimes in a way they didn't expect.
Proximate cause: buy vanilla ice-cream
Root cause: vapor lock
The letter didn't assert that the ice cream was the root cause, but made it very clear it was the proximate cause.
The Pontiac President, and the person who wrote that "moral of the story", may have confused the two. But the engineer in the study didn't.
I used to do some help desk as part of my dev job, and from my experience, users easily assign any random fact as the source of the problem. Often things like "correlation is causation" or *post hoc ergo propter hoc" (after that, therefore because of that). Good as heuristics, but bad when they are substitutes for reasoning.
Users cannot diagnose at all because they have no idea of how the thing they are using works (which often normal - we are the engineers, they are the users) (in this regard, users that think they know "that stuff" are among the most difficult to deal with). One cannot properly diagnose something one doesn't understand well.
We got the wiper fluid filled, so the mystery is in remission, but I'm wondering if all warnings will pop up right then. I'm guessing it has something to do with the telemetry of the car being nudged in that spot, waking up and saying something.
Could be giving just enough weird EM interference to bump the sensor from "enough fluid" to "low"
(I know I sound like Scully from x-files but could just be?)
The road is like that for miles with some lights, traffic circles, etc... Only warnings in one spot.
I really didn't believe my wife, or thought maybe it was happening a few times when she made a regular trip like getting drive thru coffee and coming right back. But then we ran a bunch of errands all afternoon and got one warning there on the way out, and one warning on the way back.
That’s probably it.
If I were designing a wiper fluid warning, I would use some sort of fluid level sensor and I would denounce it aggressively: the indicator would only light up if the sensor detected a low level for more than a couple seconds. That way traffic circles, bumps, etc would not cause many false positives. I might even couple it to some kind of acceleration sensor so a warning would not turn on during or shortly after any heavy vibration or acceleration.
A long slow bend would cause a prolonged, steady centrifugal force and/or sideways acceleration due to a banked road, which would defeat these mechanisms.
If you are making a turn the centrifugal force will push the fluid away from the axis of rotation. There is likely a level sensor only on one side of the tank, so the turn might push the fluid away from the sensor enough to trigger a warning.
The sensor likely has a time component to avoid triggering every time you make a turn, but if this bend is long enough, maybe the fluid is displaced long enough that it overcomes that minimum time.
Maybe you found the sweet spot at a certain point in that long, slow bend?
He explained: once upon a time, the machine refused to run _any_ Java programs, and would spectacularly crash and burn instead. C++ fine, python fine, anything Java was a hard nope. He didn't believe this at first until his program also started crashing the machine.
It took a tech, him, and another professor about two weeks to work out that the JVM happened to allocate the same RAM address to the integer 12 on that particular machine every time the JVM started. The actual chip of RAM that contained that hardware address was faulty, so whenever the machine tried to allocate to that address, it would crash.
Swapping out the bad RAM stick immediately solved the problem.
I won't spoil the conclusion. It's a long and winding path to get there and a good read. HN discussion:
> In causal inference, a confounder (also confounding variable, confounding factor, extraneous determinant or lurking variable) is a variable that influences both the dependent variable and independent variable, causing a spurious association. Confounding is a causal concept, and as such, cannot be described in terms of correlations or associations. The existence of confounders is an important quantitative explanation why correlation does not imply causation. Some notations are explicitly designed to identify the existence, possible existence, or non-existence of confounders in causal relationships between elements of a system. / Confounds are threats to internal validity.
Here is a sketch of a statistical model that shows a confounder (a variable affecting both the dependent and independent variables)
Explanation: `T` influences `S` because a shorter time leads to `H` (a hotter engine, which is prone to vapor lock). And `T` also influences `I` (type of ice cream chosen) because the placement of vanilla ice cream allows for quicker purchase. Voila, now we have a spurious relationship between `I` and `S`.
S = f(H, I, T) `S`: car starting or not (dependent variable) `H`: how hot is the car engine (independent variable) `I`: ice cream type chosen (independent variable) `T`: time taken to buy the ice cream (a confounder)
I'll start with (c). Attempting to talk about a model in isolation from its experimental design can be misleading, as it ignores the context that gives the model its interpretive power and validity. In this case, a good experimental design must include a sufficiently diverse sample of people to account for variation.
Regarding (b), depending on the person, the influence could flow either way between `I` and `T`, to varying degrees.
- Example of `I->T`: One person might come into the store strongly preferring one type of ice cream (`I`) and be willing to take time to look for it (`T`)
- Example of `T->I`: Another person might come into the store in a hurry and be motivated to procure the closest ice cream flavor.
Regarding (a), no model is 'true' but some are better than others for particular purposes.
- To the extent that prediction is the key goal, confounding variables don't usually matter.
- But to the extent that _statistical inference_ is the key goal, there are many techniques for teasing apart influence.
Unfortunately, too often in machine learning contexts, the word "inference" refers to the process of using a trained model for _prediction_. Yikes. This contrasts sharply with the term's use in statistics. The field of statistics got this one right, even as ML techniques have taken off spectacularly.
The electrician came out to check it and gave me the most incredulous look when I told him it didn't work at night. But he went to take a look. Came back later and said that the wire was barely touching the light fixture. So, at night, when it go colder it would slightly pull back and no longer be touching. During the day it would warm up, expand and would work just fine.
haven't figured out a solution, i just run a drip that is loud enough i notice if it stops so i can go out and reseat the wire, otherwise the stuff outside the well can freeze and that costs money to replace.
Knowing that this was bullshit, I tried everything else to no avail. I finally caved and slammed the glove compartment. To my surprise, Bluetooth has been solid ever since...
Finally, I looked at her mouse: it was half of a wireless mouse-keyboard combo. Turns out, the keyboard hadn't been turned off before it went in the pile, and other keyboards on top of it were pressing its keys, or rather just the one key.
A month into summer, the internet started to die at around 1pm… then magically restore itself in the early evening before the tech would arrive. This went on for a solid week or more.
Eventually the provider agreed to send a tech immediately after we called. On first inspection, everything looked good. Thankfully he was diligent and found that an old pin based IC was likely expanding in the heat and every so slightly unseating itself in its socket. Properly seating it and adding some hot glue solved the issue.
Never underestimate heat dissipation in product design.
A DisplayLink KB article even mentions it (and the associated white paper about the issue), stating:
Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue.
DisplayLink Article: https://support.displaylink.com/knowledgebase/articles/73861...
Direct whitepaper link (warning: PDF): http://www.emcesd.com/pdf/eos93.pdf - if people prefer to search themselves and not use my direct PDF link - it is entitled "A New Type of Furniture ESD and Its Implications" by Douglas C. Smith, from 1993
Went to a friend's house years ago, and knocked on the front door. Loud running, then BOOM door moves as dog jumps up onto other side. After some finagling, my friend finally gets the door open and says "Use the doorbell next time, he hasn't figured it out"
My local grocery store has a selection of ice creams in an end-cap cooler at the face of an aisle. The complete frozen goodies selection is down at the back end of the store. I almost always snag what I want from the up front cooler rather than making the trek to the back.
Plus the hand-held frozen goodies, like ice cream truck fare, are usually in their own cooler separately from the big stuff.
Over time they ended up building homes in this manner too and one of the brothers purchases the largest available plastic watertant on roof. Offended the other brother decided that he will build even bigger water tank. So they built a massive concrete water taken on the roof.
However the problem with this tank was that it would simply not retain the water which was pumped into it via a small motor pump which pulled the water from nearby well.
This resulted into the brother accusing the brother of "black magic" and engaging in daily fist fights and abuse.
Eventually someone figured out that the person who connected the intake pipe connected it at the bottom of the tank. So wehn the tank was full the water would simply go back to the well.
After some time, I noticed that the phone seemingly only crashed in one area of the open office floorplan where I was working.
I started walking around the office testing this theory, not really believing it. But after a while, I had hard evidence that the bug would only manifest once I entered that part of the office.
When I came to terms that I wasn’t hallucinating, I realised what the problem was. There was poor reception in that part of the office, causing the phone’s modem to switch from 4G wideband to narrowband (glossing over details here), which triggered the bug.
Easy to see with hindsight, but I was very confused there and then
Randomly during longer trips, the car will just die for no discernible reason. It's the Car of Theseus at this point with how much I've replaced, but the issue persists, and the nature of these intermittent problems makes debugging a nightmare. More puzzling still that the car starts up fine after a short nap.
I had a similar issue that turned out to be a slightly loose battery connection. While the battery clamp was making contact with the battery terminal, I didn't tighten it enough and it made poor contact.
Is it correlated to temperature at all? If it is, I wouldn't be shocked if something, like a relay, is building up heat and increasing resistance to the point of operating incorrectly. A short nap might give that component enough time to cool off.
After a while, I correlated the problem to very high humidity: usually happened during heavy fog or rain. So, it's probably an ignition problem, right? Replace spark plugs. Nope. Distributor cap/rotor. Nope. High performance plug wires. Nope.
Drove me nuts for about two years. Then one day I'm in my garage looking for something and I move my timing light out of the way. Hmmm, didn't think about that...
After two years, problem turned out to be timing slightly out of spec. Fixed in five minutes!
The particle accelerator would start overheating every day right after lunch time. They eventually figured out that enough people were using the bathroom after lunch that it was affecting the water pressure in the cooling system!
I'm not very good at retelling stories.
I don't know whether this is true or not, but either way: incredible. Love this, and love the company for actually sending someone on company time (!??!?!?!?) to check it out. <3 :'))))
This feels very contrived; who organises a supermarket like that? "Where's the ice-cream?" "Which flavour? We keep them separately, to make an anecdote work."
(Like, the moral of the story still works, but the specifics feel very dubious...)
I could def. see Vanilla Ice Cream being in that container for a while for whatever reason.
Or it could be a small/local grocery store and the owner/manager really likes Vanilla, and did it for their own convenience.
Or I could imagine something like a Sonic drive-through, where you often pull up in to a slot and turn off your car, but only until your order gets brought to you by a bellhop:
- works when I order a burger (burger takes a few minutes to cook)
- doesn't work when I order a hot dog (those are hot and ready to go)
or something like, works when I order a chili dog (because they have to heat up the chili)
.... Wait, that's still a thing? Not American, and my visits there have only been to urban areas, but I had kind of assumed that this was something present in films from the 1960s, not, like, 2023.
At the very least, chocolate is neck and neck and on the only sales chart I could find (which I think was for the UK?) chocolate sold more than vanilla.
Is it is an understatement to suggest this is a highly unlikely circumstance?
Everything seemed to be going well, but then we hit a snag. Our server began restarting unexpectedly at random intervals throughout the day. Initially, we were baffled. We assumed that the application was crashing because it could not handle the load, and we spent hours digging through code and logs.
But the real culprit turned out to be something entirely different. The server was located next to a storage cabinet, and its power supply line was inconveniently blocking the cabinet's door. Whenever the nurses needed to access the cabinet, they would simply unplug the server, get what they needed, and then plug it back in, oblivious to the chaos it caused in the system.
It taught us the importance of considering all variables, even the human ones, when troubleshooting technical issues. And so, we moved the server, cleared the path to the storage cabinet, and things were back to normal :)
On the other hand, my previous Honda Civic with its dual voltage electric system (computer thinks the battery needs charging? Generate 14.4V. Computer thinks it doesn't? Generate 12.6V) caused us considerable grief until we just started driving around with the headlight switch on all the time (this forced it into the higher charging voltage). This "feature" is not well known/understood even by mechanics and has probably caused untold numbers of alternator replacements.
Current Civic is so automated that even the headlight on and high/low beam is under the computer's control. Hopefully no weird chronic computer bugs in this one.
Although the AC and car fans do overtime here.
The key piece of evidence was when we moved a printer into his office, and it had the same disconnects, at the same times. Every day around midday. Not one disconnection, mind you - starting at about 11 the disconnects would get more and more frequent. From noon to one it was every few minutes. In the afternoon and evening it was just fine. It just so happened that the engineer would work in the morning, and his first compiles if the day were around that time. So, compiles were a red herring.
In the end the problem was the kitchen one floor down from his office, where a microwave was heating up peoples' lunches and causing enough EMI to disrupt (in those days rather brittle) wifi. We bought them a new microwave, and the disconnects stopped.
Car allergic to vanilla ice cream (2000) - https://news.ycombinator.com/item?id=13347852 - Jan 2017 (133 comments)
and moving a controller crashes the PS1.
Makes me thinks there are tons of these issues out there.
I decided to shelve a few extra bucks and purchased a Camry from the same dealership. The second day while I was parking my new Camry, I found the same issue of the steering wheel getting stuck all over again, with this new car. So this was surely a me-problem, and not the car problem. After some analysis I relaized that I was turning off the car at red lights, without shifting to park. So when I turned it back on, the steering would lock itself. But if I shifted to park first then it would work perfectly!
He was a flamboyant character and had a habit of talking loudly on the phone for an hour or so a day. I modified his mouse driver to give me remote control on demand and installed it when he was away from his computer.
For the next few weeks, I would wait for his daily call and gradually move his mouse as he was trying to use his computer. He was visibly frustrated and eventually caught me when he went on a rampage during his conversation and he caught me laughing.
However, when driving around with our test car, from time to time the engine stalled, or stalled an restarted. This seemed to appear randomly, and since the engineer was busy with doing the engine application, he asked me if I could try to take care of the issue.
I noticed that the problem occured only during rain, then I noticed that it only occured when the windscreen wiper was set to interval. With some help from engineers from the main office, I was able to find the relais controlling the interval mode for the windscreen wiper as the root cause. I was sending voltage spikes into the cable tree, causing a reset in the central control unit. A decision was made to change the supplier for the relais (this was not supplied by our company), and the problem disappeared and never occured again. Everybody was happy and I felt really proud that I was actually able to contribute to the success of the project.
the moral of this story, the engineer being a logical man refused to try reproducing the problem and devoted infinite time to watching the user shop every night. that way he gets paid to hang and eat ice cream
Relevant thread: https://news.ycombinator.com/item?id=23005140
I got to site and found a loose connector on an IO module under the operator's movable arm rest. It was the connector that carried the CAN comms. I plugged it in. No more dramas.
I can only surmise that the process of them slamming the arm rest up, tromping down the stairs, flicking the isolator off and on, tromping back up the stairs, and slamming the armrest down was enough to re-seat the loose connection temporarily.
She began to complain that it was locking up. It happened to her over and over, but not to me, so I wasn't sure why.
One day I had occasion to remove the case and look inside. The CPU fan had become dislodged, and would spin uselessly without cooling down the fast 486.
I owed an apology to my girlfriend. Many years later, I was able to apply this elementary knowledge to help out my father, whose notebook always locked up. He put it on a cooling pad, which was enough to allow the vents to work, even without running those external fans.
One day I started to understand what’s happening when I touched the aluminum Apple keyboard while being electrically charged. I was wondering how it was possible to get an electric shock since my Mac Mini had no connection to ground via its IEC-60320 C7/C8 connector. I learned that the Mac Mini grounds itself via DVI.
Turned out the pressure cylinder of the chair caused some kind of electromagnetic pulse, which interfered with the DVI signal and forced the monitor to resync every time I sat down.
Ice cream is not that popular, so it could reasonably be stocked near the front of the store. But all the flavors are stored together, as you'd expect.
Engineers who make house calls are also not a thing in Detroit. You might see that happen in Japan, where airline CEOs have been known to make pilgrimages to call on the families of crash victims. But I'd be (pleasantly) surprised if the car companies have ever done anything like that.
Milk and eggs are at the back of the store because they're high-turnover items that require refrigeration. In literally every grocery store in my area --- Pete's, Whole Foods, Caputo's, and Jewel --- produce and high-volume soft drinks are both in the front of the store; in fact, the major soft drinks are stocked outside the store at Pete's, in the vestibule where the shopping carts are.
In no grocery store I have ever been in have the bulk of the milk and eggs been in the front of the store, despite the fact that stores compete with each other, and are a low-margin business.
Grocery store layouts are certainly optimized. But I don't think the "milk and eggs are hidden so you'll buy snacks" narrative makes much sense. The stores are loaded from the back, and easiest to refrigerate from the back, and having clerks constantly trucking milk from one refrigerator in the back to another in the front of the store seems pretty suboptimal. I think dairy placement is a constraint the stores work around, not an evil scheme.
> Ice cream is not that popular, so it could reasonably be stocked near the front of the store. But all the flavors are stored together, as you'd expect.
This part is actually more plausible than you'd expect, especially if you assume that details are slightly garbled by the N'th retelling.
Last year, we went to a convenience store to pick up some ice cream treats after a big hike. There was one of those mini freezers with a set of frozen treats right next to the entrance, facing the cashier. But there was also a line of full-sized freezers containing all the usual frozen goods (including frozen ice cream treats) in one of the aisles. Depending on which frozen treat you wanted, you'd either be picking it right up by the front door, or having to wander down an aisle to find it.
Ice cream tends to be somewhere close to the middle of the store. But if you're after staple foods in larger-than-snack-size quantities, you'll be doing some walking.
(It's an HN perennial.)
The included details of the narrative are deliberately fanciful to make it obvious that it’s not intended to be taken literally.
Of all the details people are trying to pick apart, I’m surprised nobody mentioned how strange it was that they drove to the store every single night for ice cream rather than just buying a few large containers and putting them in the freezer. :)
The next day there was an engineer sitting in the back of my car with a bunch of test devices capturing traces of the BT comms with the head unit. Apparently Subaru didn't sell enough units to warrant its own certification process for BT so this was the first time engineers had looked at it. IIRC it did get better a few updates later; it was maddeningly unusable out of the box.
After 3 or 4 trips across the county to the dealership she threatened to invoke the Lemon Law on the new vehicle and wouldn’t you know it, Jeep sent out an engineer to Beaufort, SC and he spent a week on this vehicle. Was fixed and never had a leak again.
When I was at Amazon, Jeff sent out a question-mark email after a complaint from a customer, something he used to do when a direct email complaint caught his attention in some particular way. The complaint was that some amazon hardware was having difficulty with this customers wifi network. I want to say it was a 1st gen Echo, but it has been close to a decade since, and I only saw the report that was produced from the situation, that I honestly forget the particulars.
That question mark email ended up with Amazon sending some senior engineers around to go figure out what was going on at the customer's home, and figure out what that meant for the device and how it might be mitigated. It ended up being some weird combination of physical properties of the house, the wifi arrangement, and some suboptimal behaviour in the Amazon device that was fixed via a subsequent software update.
Vanilla is the most popular flavor? That makes me skeptical of the whole story. ;-)
An example of the top 21 flavors (in America): https://www.krqe.com/news-resources/ranking/the-21-most-popu...
I was a picky eater as a kid, yet I still can't fathom why so many kids prefer vanilla over other flavors.
Also, vanilla is good.
This is the only slightly credible bit IMO; companies sometimes do this! If you have a really weird problem in something you've sold a few million units of, you really, really want to know what that problem is, before more people start complaining.
The supermarket layout is clearly contrived to make the story work, though, and doesn't otherwise make any sense.
I will never believe that GM sent an engineer to troubleshoot this problem. Maybe in a small town a local dealer's mechanic spent some time troubleshooting but he probably figured out it was based on the short shut off time. Then he speculated that it took less time to get Vanilla than the other flavors? I doubt that is true - most likely it was random. It is amazing how often a random sequence will have an extended sequence that our brains won't believe is random.
Not sure I even believe there was any genuine observation of correlation from the owner. These type of apocryphal technical support stories are almost always formed backward from a problem and someone tries to think of the most insane way it could have manifested. But people love to believe them.
The likeliest correlation is that I have the Pi dump backups at that time, and it may be crashing the network stack due to unexpected hardware output because running a hard drive and the internal wifi simultaneously under-volts the system. But it sure does look like it just gets visited by demons in the pre-twilight hours.
Incidentally, I still don't believe a word of this story (at least as it's told here). The short delta of time difference between walking _further_ into and out of a store would not have enough impact on the cooling of the engine to make a such a substantial difference as to it starting or not. It simply will not bleed off that much more heat unless this store is a mile long and it's an additional 20 minutes to get a different flavor.
The only reason I express the doubt over it is because it makes the story a contrivance, which makes it pointless. If the person's different activities _actually_ resulted in a significant difference of time the car has been sitting, it's likely the owner themselves would be able to easily deduce what could really be the issue. By pretending the issue introduces some very small delta of time, it arbitrarily masks the true cause (which is the entire point of the story).
Oh, I wouldn't even remotely bet on that. Even in my professional sphere of programming I've been caught by what I call "cognitively available" theories of what the problem is that turn out to be entirely wrong in the end, because the real problem is something I wasn't even remotely considering before hand, and possibly even would have dismissed if it had crossed my mind.
If you don't even know what "vapor lock" is, and I assure you this will describe the majority of car owners, why would you think "time in store" is the difference?
What is cognitively available to this person is that they buy different sorts of ice cream and that causes the problem. It puts the spotlight of cognition on that factor to the exclusion of other things. Even the engineer trying to solve the problem was probably slowed in his investigation by such an appealingly available issue being proposed first; again, I've certainly experienced this in my own professional sphere where someone proposes some explanation that ultimately turned out to be completely spurious, and it takes actual effort to get both myself and my team off of that line of thought.
I had a brand new AC/heater hvac system installed last year. Encountered this problem, the AC would just turn off. First time, called the AC company and they come out. After a few visits, they pinpoint the problem as the condensation pump, it’s just a small water pump that’s down in my basement to push water out a little PVC pipe. When the water fills up in the reservoir, a pump kicks on and moves water. If the water gets too full, a sensor turns the AC off. After the rd visit they decide that “the grade isn’t gradual enough and the pump turns off”. They replace the pvc. Fourth attempt they replace the pump, pump must have failed. Fifth visit, they replace the pump with a different brand. Sixth visit they disconnect the automatic shutoff of the AC, my basement floods with water. Seventh call they change the pvc pipe again. Eighth call they are baffled, say everything is working as it should, give it time.
After they leave, my wife goes “whenever I turn the basement light on, I hear a weird noise”. I immediately turn the switch off and walk down into the basement and test the outlet. The light oddly enough controls a random outlet the pump was connected to. Every time a tech came over, they turned the switch on.
I still wonder how many more phone calls it would have taken before they figured it out.
One day, I dropped by to say hi, but they were both out. I decided instead to take the car for a spin around the local beachfront to give it a turn over. A few minutes into the drive, I started to lose engine power, so I pulled over. The engine then completely died on me, so I let it sit for a moment before trying to kick it over again.
The fuel pump is such that when you first turn the key to power the accessories, you hear it go tickticktick tick tick.. tick... tick...... - the ticks slow as the pump builds up fuel pressure. It's a good audio cue as to when you can then turn it to ignition and kick the engine over. In this case though, the ticking wasn't slowing - just the same tickticktickticktick. I tried to kick it over several times, but no matter which deity I invoked, no luck.
Empty fuel tank then. I checked the fuel level (walk to the tank on the back and poke a special bit of wood in to see how full it is) but lo and behold, plenty there. So I let it sit for a few minutes more and try again, hoping it may be to do with a flooded carburetor after my several attempts to restart it. The same: tickticktickticktick.
I gave Dad a call, describe the problem, tell him what I'd done to solve it and that I'd concluded the fuel pump must be cactus. I asked if he was going to be home soon to come give me a tow home. At this point, I learnt that he was some hours away and Mum was overseas, so no luck there. As I'm mentally preparing to push the car home for over an hour, he interrupts - "open the bonnet, grab the spanner out of the toolbox there and give the pump several hard whacks".
"Beat the bejeezus out of the fuel pump a few times."
So I did, and I turned back on the accessories. Ticktickticktick ticktick tick tick... tick.... tick.......
Dad then explains that this problem's been around for decades. Very occasionally, a bubble of air will end up in the fuel feed line. It then blocks the pump, which can't clear it, but a bit of suitably percussive maintenance consistently dislodges it and the pump can draw fuel in again.
As they say, old cars definitely have character, and I think that comes about largely because people can understand, fault-find and fix these sorts of analogue issues that arise. New cars are much more reliable and don't face nearly as many random faults, but those that do happen are almost impossible for Joe Public to resolve on the side of the road.
Disclaimer: I worked for a startup named after this bit of folklore.
Took me straight back to University ...
The President/CEO would not have been given this to decide on, for one thing.
... if it were possible to prove that some event like this never happened, I'd put money on it.
my father would have eaten this story up and would have "improved" the tale, and then written it down for others, who would have all done the same.
How was that the only occasion he tried to start his car that quickly after shutting it down?
Why did the family always buy ice cream by itself?
One day a coworker got a support call from a user. Far from mythical outbursts of irate customers, an understandable the voice at the other end of the line remained extremely calm and eager to help resolving the issue, although you could feel a gleam of despair in the back of her voice.
The woman at the other end of the line had to input troves of data daily in a specific form in the application, basically taking a stack of whatever field-ridden paper forms they received and entering them into the system via the equivalent digital form manually.
She was adamant that every time she attempted to input a specific, optional date the form would close on its own, losing all of the input data. To make matters worse the specific date was the last one to be entered, thus all the already-repetitive hard work of inputting an entire form was getting even more annoying.
To make matters even more complicated, she was working part-time, in rotation with another person covering for days she was off. The other person never encountered the problem!
So their solution for years had been that the woman who was reaching out to our support would input everything but the affected date field, save but not validate the form, and place the paper form in a special basket; the second person would then go through the paper stack again and fill the missing dates.
They would go to the extent of telling people who submitted the form that the affected date would lengthen the form processing by a few days. Somehow this flied quite well as people would understand that this was a case where the date would imply more work (as in, actual work, e.g cross-checking more stuff or more internal paperwork or whatever) on our customer's side.
In any case, this solution would stop working soon as the woman was going to go full time again and the other person in rotation would move to do something else. The customer had put up with this for an absurdly long time but now they were painted into a corner so they were left with hardly any option but to reach out for support.
We couldn't hardly believe this was happening, but the person was very articulate and came up with a quite complete description of the situation that in most support cases you only dream of having. So my coworker dove into the code base, which was not ours and only received the occasional bug fix or small development to keep the software compliant. He could not find anything in the code, he could not reproduce anything locally, the date field - of any of the other fields for that matter - looked like any other standard field, but there must have been something.
Confounded, the only thing we could think of was that it only happened on the user machine somehow. He asked to remotely connect and try it on the user's machine. No dice. He asked for the person to do it, anxiously watching as the user cautiously moved the pointer towards the date field when suddenly the form vanished! He went at it again and did the exact same thing, down to when things are clicked or typed and whatnot, but nope! But the user could still reliably reproduce it, over and over again.
Could it be that doing things through a remote connection affect the ability to reproduce? Out of options, he asked for the other person - which likely was at the office that day - that used to input the form to come over and input the same thing directly, watching the screen as fields filled up. It looked _exactly_ the same as when the usual user was doing it, but it worked.
We now could not see any other option but to move the investigation on site. Two people were tasked to that end to cover all possible grounds and brainstorm on the spot.
Arriving on site, they were greeted by this very amenable woman affected by the bug. They entered a small, poorly lit office, one taking a seat next to the user and the other standing due to the lack of space, and watched the user as she powered up an absurdly small, overaged CRT screen, then booted the ancient PC. As the user started typing on the $5 keyboard, moved the creaky mouse around they could not help but feel compassion as they saw the daily stack of paper forms that had to be entered every day, all day long, using such poor equipment and inhumane conditions when the customer company was throwing seemingly endless and voluminous amounts of money at our employer.
Focusing on the actions on the screen, the team member leading the investigation went through all stages of bewilderment and despair as the user arched forward to aim at the impossibly tiny date field on that tiny old screen, and just like that, the window vanished. His mind helplessly racing, in the stillness of the moment a voice raised up from behind:
Said the one standing up.
"Can you do it again?"
So, with unabated calmness only zen masters can achieve, she nodded, proceeded with moves repeated a million times over, and the window closed again.
You see, one of the requirements for this digital form was to match the paper form in layout and wholly fit in the screen. To achieve that, the font size and UI element dimensions were dialled down. On the user's small, poor, aging screen in a badly lit environment and with an impossibly bad mouse this made UI elements spectacularly hard to interact with, especially the date picker drop down. So when reaching to that specific optional date picker in a specific place of the screen the user had to lurch forward a bit to take a good look and lurch a little bit further and to the right to extend the arm to mover the pointer over there and click to make the date picker appear. As it turns out the affected user was a bit overweight and with this very movement her - ahem - "sizeable" right breast ever so slightly brushed against a key on the right side of the old, mushy keyboard, in a way that made it register a keypress without any audible or haptic feedback. That keypress - which I can't recall which it is - turned out to be what closed the form.
"Yup, I got it".
We could have patched the software to prevent that from happening. We didn't. Instead a report was produced that in no uncertain terms - but not pointing at the actual mechanical details - a combination of hardware and poor, non-ergonomic work environment was identified as the root cause; and not just that, but through small talk it was learned that the reason she was working part time was medical and largely caused by these terrible work conditions, so we added a note "as a courtesy" they could maybe end up being liable for any health issue that would "hypothetically" come up buy working in such environments. She got a new computer and display, a proper keyboard an mouse, a better desk and chair, and, I seem to recall, an additional light. The customer complied, reported the issue as fixed, and as far as we know the issue never occurred again.
'Twas back in the early days of the internet, when dial-up connections ruled the land and floppy disks were the height of data storage. I found meself workin' as a young software engineer for a company that aimed to revolutionize the way folks communicated. They called it "email."
Now, ye might think email's as commonplace as a pot of tea these days, but back then, it was like magic. Me team and I were tasked with creatin' the very first email client for personal computers. We had grand ideas, but, oh, the challenges we faced!
Our office was a chaotic mix of wires, half-eaten sandwiches, and programmers hunched over their keyboards, with cups of strong coffee never more than arm's reach away. We'd spend hours debuggin', tweakin', and scratchin' our heads, tryin' to make sense of the code.
But there was one particularly ornery bug that had us stumped. Every time a user sent an email, a gremlin in the system would gobble it up, and it never reached the recipient. We dubbed this foul creature the "Email Bandit."
We tried every trick in the book, but the Email Bandit remained elusive. The boss was near despair, and me colleagues were ready to throw in the towel. But ye see, I had a plan. A plan so bold, it was downright cheeky.
I reckoned that if the Email Bandit was stealin' our precious messages, maybe he had a hankerin' for a particular type of email. So, I composed a message that read, "Dear Email Bandit, we've got a special treat for you. Please don't eat this one."
We sent that message out into the digital wilds, and we waited. Lo and behold, within minutes, the Email Bandit struck! He gobbled up the message and vanished into the ether.
Now, remember, we engineers are a clever bunch. We'd embedded a little tracker in that email, and it led us straight to the Email Bandit's hideout – a rogue line of code that no one had noticed before.
With the Bandit cornered, we rewrote that line, patched up our software, and celebrated like we'd won the lottery. From that day on, our email client worked like a charm, and we'd slain the Email Bandit once and for all.
So, me friends, never underestimate the power of a bit of creativity and a dash of cheekiness when it comes to solvin' software engineering conundrums. And remember the tale of the Email Bandit, a reminder that sometimes, the most elusive problems have the simplest solutions, hidden in plain sight. Sláinte to the world of software engineering!
I immediately went to time, location, etc.
Long story short, an idle timeout variable had been set to 0 milliseconds, so any connection that took 1+ ms failed, so you could only connect to systems within 0.5 light-milliseconds of the university, which is about 100 miles.