Kubernetes Nodes need occasional maintenance. You could be updating the Node’s kernel, resizing its compute resource in your cloud account, or replacing physical hardware components in a self-hosted installation.

Kubernetes cordons and drains are two mechanisms you can use to safely prepare for Node downtime. They allow workloads running on a target Node to be rescheduled onto other ones. You can then shutdown the Node or remove it from your cluster without impacting service availability.

Applying a Node Cordon

Cordoning a Node marks it as unavailable to the Kubernetes scheduler. The Node will be ineligible to host any new Pods subsequently added to your cluster.

Use the kubectl cordon command to place a cordon around a named Node:

$ kubectl cordon node-1
node/node-1 cordoned

Existing Pods already running on the Node won’t be affected by the cordon. They’ll remain accessible and will still be hosted by the cordoned Node.

You can check which of your Nodes are currently cordoned with the get nodes command:

$ kubectl get nodes
NAME       STATUS                     ROLES                  AGE   VERSION
node-1     Ready,SchedulingDisabled   control-plane,master   26m   v1.23.3

Cordoned nodes appear with the SchedulingDisabled status.

Advertisement

Draining a Node

The next step is to drain remaining Pods out of the Node. This procedure will evict the Pods so they’re rescheduled onto other Nodes in your cluster. Pods are allowed to gracefully terminate before they’re forcefully removed from the target Node.

Run kubectl drain to initiate a drain procedure. Specify the name of the Node you’re taking out for maintenance:

$ kubectl drain node-1
node/node-1 already cordoned
evicting pod kube-system/storage-provisioner
evicting pod default/nginx-7c658794b9-zszdd
evicting pod kube-system/coredns-64897985d-dp6lx
pod/storage-provisioner evicted
pod/nginx-7c658794b9-zszdd evicted
pod/coredns-64897985d-dp6lx evicted
node/node-1 evicted

The drain procedure first cordons the Node if you’ve not already placed one manually. It will then evict running Kubernetes workloads by safely rescheduling them to other Nodes in your cluster.

You can shutdown or destroy the Node once the drain’s completed. You’ve freed the Node from its responsibilities to your cluster. The cordon provides an assurance that no new workloads have been scheduled since the drain completed.

Ignoring Pod Grace Periods

Drains can sometimes take a while to complete if your Pods have long grace periods. This might not be ideal when you need to urgently take a Node offline. Use the --grace-period flag to override Pod termination grace periods and force an immediate eviction:

$ kubectl drain node-1 --grace-period 0

This should be used with care – some workloads might not respond well if they’re stopped without being offered a chance to clean up.

Solving Drain Errors

Drains can sometimes result in an error depending on the types of Pod that exist in your cluster. Here are two common issues with their resolutions.

1. “Cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, or StatefulSet”

This message appears if the Node hosts Pods which aren’t managed by a controller. It refers to Pods that have been created as standalone objects, where they’re not part of a higher-level resource like a Deployment or ReplicaSet.

Kubernetes can’t automatically reschedule these “bare” Pods so evicting them will cause them to become unavailable. Either manually address these Pods before performing the drain or use the --force flag to permit their deletion:

$ kubectl drain node-1 --force

2. “Cannot Delete DaemonSet-managed Pods”

Pods that are part of daemon sets pose a challenge to evictions. DaemonSet controllers disregard the schedulable status of your Nodes. Deleting a Pod that’s part of a DaemonSet will cause it to immediately return, even if you’ve cordoned the Node. Drain operations consequently abort with an error to warn you about this behavior.

You can proceed with the eviction by adding the --ignore-daemonsets flag. This will evict everything else while overlooking any DaemonSets that exist.

Advertisement

$ kubectl drain node-1 --ignore-daemonsets

You might need to use this flag even if you’ve not created any DaemonSets yourself. Internal components within the kube-system namespace could be using DaemonSet resources.

Minimizing Downtime With Pod Disruption Budgets

Draining a Node doesn’t guarantee your workloads will remain accessible throughout. Your other Nodes will need time to honor scheduling requests and create new containers.

This can be particularly impactful if you’re draining multiple Nodes in a short space of time. Draining the first Node could reschedule its Pods onto the second Node, which is itself then deleted.

Pod disruption budgets are a mechanism for avoiding this situation. You can use them with Deployments, ReplicationControllers, ReplicaSets, and StatefulSets.

Objects that are targeted by a Pod disruption budget are guaranteed to have a specific number of accessible Pods at any given time. Kubernetes will block Node drains that would cause the number of available Pods to fall too low.

Here’s an example of a PodDisruptionBudget YAML object:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: demo-pdb
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: my-app

This policy requires there be at least four running Pods with the app=my-app label. Node drains that would cause only three Pods to be schedulable will be prevented.

The level of disruption allowed is expressed as either the maxUnavailable or minAvailable field. Only one of these can exist in a single Pod Disruption Budget object. Each one accepts an absolute number of Pods or a percentage that’s relative to the total number of Pods at full availability:

• minAvailable: 4 – Require at least four Pods to be available.
• maxUnavailable: 50% – Allow up to half of the total number of Pods to be unavailable.

Overriding Pod Disruption Budgets

Pod disruption budgets are a mechanism that provide protection for your workloads. They shouldn’t be overridden unless you must immediately shutdown a Node. The drain command’s --disable-eviction flag provides a way to achieve this.

$ kubectl drain node-1 --disable-eviction

This circumvents the regular Pod eviction process. Pods will be directly deleted instead, ignoring any applied disruption budgets.

Bringing Nodes Back Up

Once you’ve completed your maintenance, you can power the Node back up to reconnect it to your cluster. You must then remove the cordon you created to mark the Node as schedulable again:

$ kubectl uncordon node-1
node/node-1 uncordoned

Kubernetes will begin to allocate new workloads to the Node, returning it to active service.

Advertisement

Summary

Maintenance of Kubernetes Nodes shouldn’t be attempted until you’ve drained existing workloads and established a cordon. These measures help you avoid unexpected downtime when servicing actively used Nodes.

Basic drains are often adequate if you’ve got capacity in your cluster to immediately reschedule your workloads to other Nodes. Use Pod disruption budgets in situations where consistent availability must be guaranteed. They let you guard against unintentional downtime when multiple drains are commenced concurrently.

With so much of the world’s richest man’s wealth tied up in shares of his electric car company, Tesla is a part of the deal Elon doesn’t want to do anymore for more reasons than just a few subpoenas. Six SEC filings (1, 2, 3, 4, 5, 6) revealed Elon Musk sold more Tesla shares over the last few days, which is surprising given Musk’s tweet saying, “No further TSLA sales planned after today,” after he sold off $8.4 billion worth in April.

Tesla investor Sawyer Merritt tweeted a count of Musk’s activity between August 5th and August 9th, showing he sold 7,924,107 shares worth about $6.9 billion. Just before 11PM ET, Elon responded on Twitter, citing the looming trial against Twitter as an explanation for the new plan.

Musk sold those shares earlier this year to help finance the acquisition of Twitter, and now that he’s trying to exit the arrangement, they’re going to battle it out in a Delaware courtroom starting on October 17th. Twitter CEO Parag Agrawal isn’t taking the bait on Elon’s challenge for a public debate, plus there’s no word of any pending settlement. Musk now says he has a reason to prepare for “the (hopefully unlikely) event that Twitter forces this deal to close *and* some equity partners don’t come through.” If that happens, Musk said, he’s trying to avoid an emergency sale of Tesla stock.

Exactly how investors will react to word that Musk is done selling (again) is unknown; however, the price of Tesla shares stayed flat at around $850 in after-hours trading. Wedbush analyst Dan Ives tweets, “Musk selling TSLA stock after he said no more sales will lead the Street to focus on chances of Twitter deal happening and Musk preparing cash portion … This stock sale will raise a lot of conversation on Street for bulls/bears in the morning.”

Marvel’s streaming shows and movies may still steal the majority of the headlines, but the DC Extended Universe is finally giving the MCU a run for its money. The DCEU’s myriad projects are stretching far beyond superhero mainstays Batman, Superman, and the Justice League, promising to include new and interesting takes on DC Comics’ enormous stable of characters.

What follows is an up-to-date list of everything DCEU-related that’s on the way: DC movies heading to theaters, as well as DC streaming movies and shows coming to HBO Max—a category that has, in the wake of Batgirl’s cancellation, gotten more and more in flux; we’ll be updating this post (as always) if other projects find themselves similarly on the chopping block, because that sure seems likely.

Last updated 8/9/2022.

What’s the next DC Movie?

DC League of Super-Pets

Release date: July 29 (in theaters)

The DC film everyone has been waiting for! Well, not really. But when you’re making movies with comic book characters, it makes sense to make films for all ages and this animated feature looks suitably silly and cute for a younger generation. Plus the voice cast—Dwayne Johnson, Keanu Reeves, Kate McKinnon, Kevin Hart, John Krasinksi, Natasha Lyonne—is top-notch. Will these characters cross over with Robert Pattinson’s The Batman? Of course not. But diversifying the superhero portfolio makes sense. Super-Pets also has some pretty interesting TV spots.

Upcoming DC Movie Release Dates

Black Adam

Release date: October 21 (in theaters)

After literally years of hype and anticipation (seriously, we tracked it), Dwayne Johnson will FINALLY join the live-action DC Extended Universe. He’ll play the anti-hero in a story that not only brings one massively powerful comic book character into the universe, but a whole slew of them in the Justice Society of America: Aldis Hodge as Hawkman, Pierce Brosnan as Kent Nelson/Dr. Fate, Quintessa Swindell as Cyclone, and Noah Centineo as Atom Smasher.

The official trailer came out June 8. You can check out new poster here. Dwayne Johnson also continues to share behind-the-scenes photos from the set. io9 talked with producer Hiram Garcia at a press event about what we can expect from the movie. io9 also had the chance to speak with Dwayne Johnson and the team behind Black Adam, you can read that piece here. Four variant covers for issues of Black Adam – The Justice Society Files have our best looks at Hawkman, Cyclone, Atom Smasher 1, and Dr. Fate yet.

Advertisement

A sneak peak of the movie was released during San Diego Comic-Con. We saw an action packed sizzle reel that revealed Amanda Waller (Viola Davis) returns in Black Adam as the person who is putting together the Justice Society of America.

Shazam! Fury of the Gods

Release date: December 21 (in theaters)

There was a moment where DC movies went from slightly disappointing to “oh, wait, this is actually good.” And that moment was 2019’s Shazam. So we’re very anxious to see what’s next for Billy Batson, the young boy who can say one word and turn into a superhero, especially now that his whole family is in on the excitement and they’re going up against characters like Hespera, a daughter of Atlas (Helen Mirren). Shazam! Fury of the Gods’ original release date was December 16, but has been pushed back five days. Christophe Beck (Buffy the Vampire Slayer) has replaced Benjamin Wallfisch as the composer of Shazam! Fury of the Gods. The trailer for Shazam! Fury of the Gods was released during San Diego Comic-Con.

Aquaman and the Lost Kingdom

Release date: March 17, 2023 (in theaters)

Director James Wan returns for the next adventure following Arthur Curry (Jason Momoa) as the newly crowned King of Atlantis. Now in his new role, Arthur will explore more of the world under the sea while also dealing with new threats from Black Manta ( Yahya Abdul-Mateen II) and others. This one is going to be big, big, big!

The Flash

Release date: June 23, 2023 (in theaters)

But The Flash will be bigger. Director Andy Muschietti has been tapped to make sense of this multiverse story that won’t just have multiple versions of Barry Allen (Ezra Miller) but multiple Batmen in Michael Keaton and Ben Affleck. Michael Shannon is also set to return as General Zod. We have high hopes for some Spider-Man: No Way Home-type reveals, excitement, and implications for the rest of the DC Universe. But we also have some concerns after Miller’s recent arrest, which will surely affect future Flash movies, if not this one. Though it looks like this won’t affect the movie as it is moving ahead as scheduled, however it seems as after The Flash, Miller won’t be returning as the Flash.

Blue Beetle

Release date: August 18, 2023 (in theaters)

Cobra Kai’s Xolo Maridueña is Jaime Reyes, a Mexican American student who one day coming home from school finds an ancient scarab that attaches to him and turns him into a superhero. Susan Sarandon will play the villainous Victoria Kord (a role originally given to Sharon Stone) a new character created for the film that shares the same last name as Ted Kord, the original Blue Beetle in the comics. Interesting! From director Angel Manuel Soto, Blue Beetle was originally going directly to HBO Max but will now be a theatrical release. That’s promising.

Joker: Folie à Deux

Release date: October 4, 2024 (in theaters)

Todd Phillips never saw his Joker movie as anything but a singular film. But over a billion dollars at the box office and an Oscar for Best Actor (for star Joaquin Phoenix) obviously changed things. We have a few nuggets of info so far: the title is Joker: Folie à Deux (French for “shared madness”), Phoenix is returning and will co-star with Lady Gaga (who’s playing Harley Quinn), Zazie Beetz is probably returning as Arthur Fleck’s neighbor, and… it might be a musical?

Advertisement

Upcoming DC Shows

Harley Quinn Season 3

Release date: July 28, 2022 (on HBO Max)

It’s been a long time since we last saw the Queen of DC in her very funny, very good R-rated animated show, but a third season has been in the works for a while now and we will get to see it this July. 

Upcoming DC Movies and Shows Without Release Dates

Wonder Woman 3

Release date: Unknown (in theaters)

Whether you loved Wonder Woman 1984 or not, there was no doubting it was a success. And, as a result, Warner Bros. got to work on making sure star Gal Gadot and director Patty Jenkins would come back for a third film. Now, when will this film happen and what will it be about? We have no clue. But eventually, it will happen.

The Batman 2

Release date: Unknown (in theaters)

Hot off the success of Matt Reeves’ long-awaited The Batman, everyone involved made it clear that wasn’t a one-and-done. Robert Pattinson’s Batman will be back. Will he face off with the Joker? It’s possible but not definitive. You’d imagine the Penguin and Catwoman could be back too. All of which is pure speculation because no one knows when the film will be out or what it’ll be about.

The Penguin

Release date: Unknown (on HBO Max)

You saw him in The Batman and now, Colin Farrell’s Penguin is getting his own HBO Max series. The timeline for the show hasn’t been set but you’d guess it’s after The Batman, so that he can rise up from second in command to become leader of the quack. I said what I said.

Advertisement

Peacemaker Season 2

Release date: Unknown (on HBO Max)

From the instant that opening credits scene rolled for the first time, DC fans and non-fans alike fell in love with Peacemaker. Well, he’s coming back, still played by John Cena and still from the twisted mind of James Gunn. There’s no word on when it’ll happen but Gunn has to finish his latest Marvel movie for mid-2023, so we’re guessing 2024.

James Gunn also confirmed to The Playlist that characters introduced in Peacemaker will reappear in future DC television projects, including an upcoming series he is writing and directing.

Green Lantern Corps

Release date: Unknown (on HBO Max)

From the all-star team of Seth Grahame-Smith, Marc Guggenheim, and Greg Berlanti comes this epic, 10-episode HBO Max series featuring the defenders of the universe, the Green Lanterns. The lead, Guy Gardner, has been cast (he’ll be played by Finn Whitrock), and updates beyond that should be coming sooner rather than later.

Doom Patrol Season 4

Release date: Unknown (on HBO Max)

The future doesn’t spell Doom for Doom Patrol—seeing as how last year, Warner Bros. picked it up for a fourth season. That means all your faves, played by Brendan Fraser, Matt Bomer, Diane Guerrero, April Bowlby, Joivan Wade, Timothy Dalton, Skye Roberts, and Michelle Gomez, will be back soon.

Titans Season 4

Release date: Unknown (on HBO Max)

The (not-so-Teen) Titans are coming back as well. A fourth season of the show, which stars Brenton Thwaites, Anna Diop, Teagan Croft, Ryan Potter, Conor Leslie, Curran Walters, and Joshua Orpin, is in the works.

Young Justice Season 4, Part 2

Release date: Unknown (on HBO Max)

Subtitled “Phantoms,” though what that exactly means isn’t clear yet, the second half of the fourth season of the superhero team-up show is on the way. When they return, Superboy (Nolan North), Miss Martian (Danica McKellar), Artemis (Stephanie Lemelin), Zatanna (Lacey Chabert), Rocket (Denise Boutte), and the rest will finally let us know why this season is called Young Justice: Phantoms.

Pennyworth Season 3

Release date: Unknown (on HBO Max)

Apparently, people watch this show, because it’s coming back.

Upcoming DC Movies and Shows Currently In Development

Ta-Nehisi Coates and J.J. Abrams’ Superman movie

Release date: Unknown (in theaters)

What’s next for Superman? It’s a big question in the current DC universe and, at the moment, there are two versions potentially in the works. One is from writer Ta-Nehisi Coates and produced by J.J. Abrams. There hasn’t been an update on this one in a while but we hope it’s still percolating.

Advertisement

Michael B. Jordan Superman Series

Release date: Unknown (for HBO Max)

In addition to the Coates-Abrams film, star Michael B. Jordan is developing a Superman take for HBO Max as a limited series that would feature Val-Zod as opposed to Clark Kent. Again, there hasn’t been an update in a while but we hope it still happens.

Static Shock

Release date: Unknown

Part of the relaunch of the Milestone comic brand is this adaptation produced by, once again, Michael B. Jordan. A writer was hired back in 2021 to come up with a script and we haven’t heard much since—but since DC made such a big deal of bringing the Milestone brand back, we’d imagine this story of a teen who can control electricity will shock theaters eventually.

Justice League Dark

Release date: Unknown (for HBO Max)

What was once looking like a possible big-screen adaptation is now, as far as we can tell, going to be an HBO Max series. J.J. Abrams is among the people involved with bringing this evil super-team to life and while there hasn’t been a ton of movement on the project of late, it’s been around long enough that we’d imagine it happens eventually.

Metal Men

Release date: Unknown

One of the more recent DC films to go into development is this tale of six beings based on various powerful metals. It’s from directors Ron Clements and John Musker (Aladdin, The Little Mermaid, and Moana), so if it happens, it’ll have an incredible pedigree.

Zatanna

Release date: Unknown

Last year, Emerald Fennell won an Oscar for writing Promising Young Woman. She followed that up landing the job to write this DC superhero flick about the young magician. Like most of the films in this section, there hasn’t been movement in a while, but Fennell is such a talent that if it happens, it’ll be something special.

Hourman

Release date: Unknown

Yes, the hero who takes a drug that makes him a hero for an hour was, back in 2021, rumored to be in development. Whether or not that’s still the case is unknown but we figured it was worth noting.

Untitled Arkham Show

Release date: Unknown (on HBO Max)

In addition to The Penguin, Matt Reeves has said he plans to expand the world of The Batman into HBO Max shows. At one point, a series about Gotham PD was in the works, but now he says it might be more about the inner workings at Arkham Asylum. Maybe. We’ll see if it comes to fruition.

Want more io9 news? Check out when to expect the latest Marvel and Star Wars releases, what’s next for the DC Universe on film and TV, and everything you need to know about House of the Dragon and Lord of the Rings: The Rings of Power.