Status updates for the Solana network, the Solana website, and the Solana Explorer. Follow
@solana
for daily announcements. Run by the Solana Foundation.
The Solana validator community successfully completed a restart of Mainnet Beta after an upgrade to 1.6.25. Dapps, block explorers, and supporting systems will recover over the next several hours, at which point full functionality should be restored.
The Solana network is experiencing an outage and not processing transactions. Developers across the ecosystem are working on diagnosing the issue and to restart the network. More information will be provided as it becomes available.
1/ Solana Mainnet Beta encountered a large increase in transaction load which peaked at 400,000 TPS. These transactions flooded the transaction processing queue, and lack of prioritization of network-critical messaging caused the network to start forking.
After an investigation by developers, ecosystem teams, and security auditors, it appears affected addresses were at one point created, imported, or used in Slope mobile wallet applications. 1/2
Engineers from multiple ecosystems, with the help of several security firms, are investigating drained wallets on Solana. There is no evidence hardware wallets are impacted.
This thread will be updated as new information becomes available.
The v1.17.31 release is now recommended for general use by MainnetBeta validators. This release contains enhancements which will help alleviate the ongoing congestion on the Solana Network.
1/ Solana mainnet beta is experiencing high levels of network congestion. This is related to issues previously identified that engineers have been working to improve and resolve, starting with the 1.8 release.
Solana Mainnet Beta lost consensus after an enormous amount of inbound transactions (4m per second) flooded the network, surpassing 100gbps. Engineers are still investigating why the network was unable to recover, and validator operators prepare for a restart.
Engineers from across several ecosystems, in conjunction with audit and security firms, continue to investigate the root cause of an incident that resulted in approximately 8,000 wallets being drained. 1/2
Solana Mainnet Beta suffered a performance degradation, temporarily reducing network throughput. The network has recovered, and engineers continue to diagnose root cause.
On June 1st, Solana's Mainnet Beta cluster suffered a four and a half hour outage caused by a bug in the durable nonce transactions feature, which resulted in stalled consensus. Read the full report below
1/ The Solana Network is currently experiencing degraded performance due to an increase in high compute transactions, which is reducing network capacity to several thousand transactions per second. This is leading to some failed transactions for users.
Validator operators successfully completed a cluster restart of Mainnet Beta at 3:00 AM UTC, following a roughly 7 hour outage after the network failed to reach consensus.
Network operators an dapps will continue to restore client services over the next several hours.
3/ The validator community elected to coordinate a restart of the network – the community is preparing a new release, and instructions will be posted in Discord.
Validator operators successfully completed a cluster restart of Mainnet Beta at 9:00 PM UTC, following a roughly 4 and a half hour outage after the network failed to reach consensus. Network operators an dapps will continue to restore client services over the next several hours.
On April 30th, Solana's Mainnet Beta cluster suffered a 7 hour outage caused by stalled consensus. Below is an investigation into the cause of the outage, and the steps core developers are taking to increase network resiliency and stability.
Resource exhaustion in the network is causing a denial of service, engineers are working towards a resolution. Validators are preparing for a potential restart if necessary.
Engineers from across the ecosystem are readying a new validator software release, which includes a patch to address an issue which caused the cluster to halt. Validator operators should prepare for an upgrade and restart of the network.
This exploit was isolated to one wallet on Solana, and hardware wallets used by Slope remain secure.
While the details of exactly how this occurred are still under investigation, but private key information was inadvertently transmitted to an application monitoring service. 2/3
2/ This forking led to excessive memory consumption, causing some nodes to go offline. Engineers across the ecosystem attempted to stabilize the network, but were unsuccessful.
An exploit allowed a malicious actor to drain funds from a number of wallets on Solana. As of 5am UTC approximately 7,767 wallets have been affected.
The exploit has affected several wallets, including Slope and Phantom. This appears to have affected both mobile and extension.
Block production on Solana mainnet beta resumed at 14:57 UTC, following a successful upgrade to v1.17.20 and a restart of the cluster by validator operators. Engineers will continue to monitor performance as network operations are restored.
The outage began at approximately
This does not appear to be a bug with Solana core code, but in software used by several software wallets popular among users of the network.
Updates will be posted to as they become available. 2/2
1/ Engineers across the ecosystem deployed several mitigations in 1.6.25 to increase network resiliency during periods of extreme transaction load. These fixes included preventing exclusive locking on accounts that reference program code
Mainnet Beta Validators: Please follow the restart instructions linked below, and upgrade 1.9.x to 1.9.28 – If you are on 1.10.x please upgrade to 1.10.23
There’s no evidence hardware wallets have been impacted – and users are strongly encouraged to use hardware wallets.
Do not reuse your seed phrase on a hardware wallet - create a new seed phrase.
Wallets drained should be treated as compromised, and abandoned.
A coordinated restart is underway to address an issue during the upgrade from 1.13 to 1.14 that caused block finalization to slow significantly.
Validators: please follow instructions below
Earlier today a bug in the durable nonce transactions feature led to nondeterminism when nodes generated different results for the same block, which prevented the network from advancing.
Validator operators across the ecosystem are working to finalize restart instructions for Mainnet Beta - if you operate a validator please follow along in mb-validators on Discord.
2/ and the introduction of throttling of forwarded transactions, which will help prevent amplification of transaction congestion.
The community is committed to publishing a detailed post mortem on the outage in the coming weeks as further fixes are deployed.
Validator operators successfully completed a cluster restart of Mainnet Beta at 7 AM UTC.
Network operators an dapps will continue to restore client services over the next several hours.
The root cause of the 2-25-23 outage is still unknown and under active investigation. The following document will be updated as new information becomes available
Engineers are currently working with multiple security researchers and ecosystem teams to identify the root cause of the exploit, which is unknown at this time.
Approximately 15 minutes ago the Solana community successfully resumed the cluster - engineers will continue to monitor network performance as network operators are restored.
Network state is secure, as are funds. Validator operators are coordinating a restart from the highest confirmed block in public on mb-validators in Discord.
An update on the August 2nd wallet application vulnerability incident.
To reiterate: if you are a user of Slope, or have ever previously imported seed phrases into Slope, your wallet may have been compromised, even if you never signed a transaction.
Mainnet beta Explorer and Solana Foundation Public RPC endpoints are currently offline as RPC node software is upgraded, following a bug in test release 1.14.
Block production has not been impacted and the Solana network has not been impacted.
On August 23, core contributors at Solana Labs asked for volunteers to upgrade their mainnet-beta nodes to v1.16.9 with the goal of getting 10% of stake running v1.16. This is standard operating procedure for upgrading mainnet-beta. Beginning upgrades with a small set of
5/ These forthcoming releases are aimed at improving the state of the network, with more improvements expected to roll out in the next 8-12 weeks. Many of these features are currently live on Testnet, where they are being rigorously tested.
RPC service run by the Foundation is experiencing intermittent issues due to high load, resulting in slow load times and unsuccessful requests for dapps that use Foundation endpoints. Blockchain itself is not impacted. Status at &
Ping tool that feeds data to is undergoing some maintenance, impacting ping analytics. This is a data tool, which has no impact on the network.
You can run your own ping test server with the following repo
RPC Services:
A new feature was activated during the last epoch boundary, which caused version 1.8 RPC nodes to fork away.
RPC Nodes that had already upgraded to v1.9 were not affected.
3/ Developers have made lots of progress on this, but the work is not complete. The last 24 hours have shown these systems need to be improved to meet the demands of users, and support the more complex transactions now common on the network.
As always, hardware wallets are strongly encouraged for all blockchain users. Hardware wallets can remain secure even if the software wallet (or the entire computer) is compromised, since all verification occurs independently and the seed phrase never leaves the hardware wallet.
4/ The current issue experienced by validators is due to excessive duplicate transactions. Engineers have just released 1.8.14, which will attempt to mitigate the worst effects of this issue.
We're investigating reports of dns resolving issues on and , both of which are run by the
@SolanaFndn
This has no impact on the Solana blockchain network
Ping stats on will of offline for an upgrade -
@ValidatorsApp
runs its own ping if you want to check ping times.
This has no impact on performance, just the display of stats on
A coordinated restart is underway to address an issue during the upgrade from 1.13 to 1.14 that caused block finalization to slow significantly.
Validators: please follow updated instructions below