Christmas 2019

Merry Christmas!

                                                  
Well I think that pretty much says it. I hope you had a wonderful Christmas with your family and extended family.  I will get to go do that this weekend since we adhere to the will of my grandmother in this family.

Work was pleasant, nice and slow, got to chat with David which was nice.  And Steven was able to help me finish getting the certificates completed on the new site.  We have moved out of https://files.silverfyre.info and I have moved to my new test bed https://hooplalabs.net.  She is a beauty and I made her 😉  Okay yes I know it sounds silly, but I take pride in my work sometimes and this was a wonderful new idea for myself.  It has my projects as I am working on them.

Christmas Eve 2019

Merry Christmas Eve!!!!!!!

I really hope that Amber enjoys her presents, I know they weren't on her list, frankly I am not even sure if she will read the big illustrated copies of Harry Potter.  But I think they are gorgeous and maybe one day she will have some reading time and can enjoy them.

But!

I loved my gifts so very much.  I will have a nice and toasty blanket to now keep me warmer at night.  My comforter I have now is pretty darn old and just isn't the same anymore.  I will need to change my sheets before I use it because while I am clean they are not.  And the shirts were oh so very comfortable, so I can start wearing those as well.

Weirdly tonight has been super slow with not much to do.  They figured out that today's earlier issue had nothing to do with our internal teams and was a automated job that borked on its own.  Okay maybe possibly related, but they should have fixed it not us, not our script.  And the whooping one chat question was for 2FA, which is mostly run by another big team not us, which means not much I can do, and even little most teams are going to do on Christmas day.

Everyone is Dead Dave - Part 3

Well, today was a bucket of fun. I hit up a few different teams about things that are supposed to break over the holidays.

Schedules seem to mean nothing to our teammates. Stefan left at 3:45 when his shift is set to go until 4:30. Ashley who is working from home showed up at 11:52 (suppose to be there at 11:30). Then messages me 4 minutes until my shift ends asking for VPN info. Yea, this is the people I work with.

Not to mention around 4:40 PM one of the primary databases nodes goes and reboots itself. Which that in itself wasn't too bad. It recovered and the owning team verified it was fine.  No, it was the second primary database that went down that started a 5 and a half-hour long Sev 2 incident. By the time 10 PM rolled around, I just sat there rubbing my head from the headache I had from sitting through that mess.

Regardless, I am excited to spend as much of my Christmas Eve with my favorite Amber. Opening presents and enjoying each other's company. See you when we are both more awake.

The Not Early Shift

So its 8:55 AM right now, yes you read that right.  I actually woke up at like 7:45 AM and this is what I get for having Slack on my phone, I see things and now I am wide awake and rearing to go, weeeeeeee. So seems some additional fun things have been discovered by our new Elastic Search replacement done yesterday.  Originally it was a IC4G issue not allowing device info to load.  And last night we discovered that it altered how one of our ticket search systems behaves.  But arguably that may be intentional and forcing people to use that tool the proper way.  So yea.....

What I thought was an issue may not be as much of an issue as an expected result.  So bleh.  In the mean time I have discovered we are making a new data center in Osaka, Japan.

Everyone is Dead Dave - Part 2

Well, today started out interestingly fun, if you can call it that.  Stefan showed up in the office for the first time in like a week (not like Philip doesn't want us to be in the office more often). So before they are leaving like 30 minutes before I tagged Erick and Stefan asking if they could fix the TV.  Of which the usual hemming and hawing and other bullshit.  And how they don't use it and the password is in Thycotic (password management tool, super-secret).  Well with enough prodding (and me installing software on a global jump host) we discovered that Stefan has access and can fix things, so yay.  Not a minute or two after that finishes, the wonderful trilling sound of dying servers enters chat. Ding! Ding! Ding! Ding! Ding! DING!

So I engage who I thought owned it Team A and was told it was Team B.  So once there they needed Team C, and I was able to determine we needed a ticket to track the incident.  I think around this point as I was spinning up the incident was when the rest of the team got with the program and realized that shit had hit the fan.  So get the incident going, get the chat room up, and email out, and message to Dan.

The fix was pretty quick overall, we received alerts at 4:09 PM and customer impact was mitigated a 4:48 PM so just about 40 minutes, but 194 accounts were impacted still.  With that all resolved (I realized I forgot to make the JIRA) the rest of the night ran pretty smoothly.  I did get a candle ordered for Amber's mom.  Only 1 because damn those things weren't cheap and the cheaper one she couldn't seem to give me a real answer and I was getting irritated (probably doesn't help that Stefan had already driven me up the wall).

I did spend a good hour or so trying to fix a new program on my laptop to work.  I am not sure how long I will keep it since it isn't as neatly integrated as I would like.  Instead of just asking me for my 2FA code, it just ignores 2FA requests or something stupid like that and rejects my login attempt.  So now I am using a personal access token, which is probably not very safe either since it has access to all of my GitHub stuff..... I might need to go adjust some settings on that.

Ooooooooo which reminds me, Amber and I get to go look at Christmas lights on Friday which will be fun, I am super excited to do that with her. Plus Six Flags on Saturday, maybe even go see Frozen beforehand.  That would be lots of fun as well.

Tomorrow will be packed for me or maybe Wednesday, haven't decided.  I need to make a payment to the CPAP company so that I can order new supplies since my deductible is completed.  But I also need to check with my new insurance from TekSystems to see if they will cover the CPAP or if I will end up with a new one since the rental agreement is with Aetna and I won't have them here in like 2 weeks.  Also need to swing by Kohl's to return Amazon items and swing by the mail center for Amber's gifts.  And I think that might just cover everything.

Maintenance

As I was prepping for tonight's maintenance, (glad I did) I found that one of the servers needed for our checks is missing the needed scripts.  So had messaged David but didn't get a response, so I ended up calling him to confirm.  Well, he walked me through moving them around which was nice and easy and now all the servers are prepped.  Now enjoy the panic of my fellow coworker:

Charles  (7:33 PM): Do you know where David's script is?
Charles  (7:35 PM): Emergency Situation
Charles  (7:35 PM): Please reply
Analicia (7:35 PM): Hit the power off switch 😛
Analicia (7:36 PM): Tis for emergencies
Charles  (7:36 PM): This is NOT a drill.
Charles  (7:36 PM): We have to run that script.
Charles  (7:36 PM): It is part of our procedures.
Analicia (7:36 PM): I am aware
Analicia (7:36 PM): and know how to run them
Analicia (7:36 PM): and have already verified they work and exist
Charles  (7:36 PM): OK, so you DO know where it is?
Analicia (7:36 PM): had to add it in to one of the servers since it mysteriously disappeared
Analicia (7:37 PM): Yea, I did this last month in November
Charles  (7:37 PM): Yes, that's great, can you please tell me where it is?
Charles  (7:37 PM): We're almost out of time
Analicia (7:37 PM): REDACTED File Location
Charles  (7:37 PM): OK, I must be on the wrong server then. It's not on REDACTED server name
Analicia (7:38 PM): dal0101, dal1001, and wdc0401
Charles  (7:42 PM): OK, we're good. My body is ready.

Glad that your body is ready, I just ran all the scripts and pasted everything where needed.  What was your reasoning for being here again?  I digress, its now 8:25 PM, 25 minutes into maintenance that hasn't started since we are waiting on networking to join the party.  And a majority of this I am going to sit here watching and fielding stupid people who can't read.  Oh, look there is networking 26 minutes late.

Maybe just maybe, we will get to start at 8:30 PM, or maybe not.  Who knows with this group.  This reminds me of school functions where nothing ever started on time because something or other ran things behind. And now the bitching about networking has begun, we are some salty people here. But can you blame them, we are 33 minutes into a window that is only so long.

41 minutes in and all the pre-maintenance checks are done which should have been finished before 8:00 PM but then again that constitutes that people show up and do their jobs.

But frankly, in comparison to last month's version of this maintenance, everything has gone much smoother.  We have made it through all of the internal steps with only a minor blip (there was a certificate mix up, which was unrelated to the changes made by the maintenance itself).  And now they are working on all the external-facing calls.  From what we have been told this means that we can do Phase 4 on Wednesday, which should complete all of the final moves of equipment out of our old Dal01.  We will have completed Exodus.

Everyone is Dead Dave - Part 1

Today has been less than pleasing, with all the extra alerts going today.  I arrived to like 10 down SOC servers.  6 hours and no one has fixed the servers.  Barely started for the day and we are already dealing with a bunch of alerts.  But what can be said except to just do the work and ignore endlessly annoying alerts. No one wants to work and so nothing gets fixed.  I am curious about what our day shift does all day since it seems not bugging teams about the alerts.

I guess of fun fact we are about to start maintenance, another I get to be apart of.  Although we shouldn't be as needed on this one since this will mostly be on the DC to do the work.  But Monday I will be responsible to assist with the maintenance with Charles.  So that will be a thing.