Did you know that with 2 Python libraries, 6 lines of code and around 15 seconds, you can load satellite data from anywhere in the world?
This is so much easier than it used to be!
Following on from my tweet about loading Sentinel-2 data, here's another example from a while ago for working with NASA's Harmonized Landsat and Sentinel data.
This example creates a cloud-free mosaic.
You do need an API key, but it's free.
Here's a new example for finding, loading and visualising Sentinel-2 data.
It includes a new `odc.explore()` function that gives you a simple visualisation on an interactive map.
Credit to
@Kirill_kzb
and
@SatelliteSci
for that new feature!
Around 30 TB of RAM and 6,000 CPUs were used to process 400 TB of data into a cloud-free mosaic over Africa. Thanks to
@whatnick
and his team, it's pretty easy to do this stuff! We'll be running this for four years of Sentinel-2 data soon in a 'real' run.
I've been working on
@OpenDataCube
and
@STACspec
integration for a while, including getting some key things into the STAC spec and now that work is paying off!
We can index data from
@Microsoft
's Planetary Computer into the ODC with a one-line command now ๐ช
More info soon...
We're getting good at this. Processing every Landsat 8 scene ever captured over Africa to build cloud free annual mosaics took two hours on a hundred spot instances.
Finished the
@LEGO_Group
world map! I didn't create a jupyter notebook and create an accurate map of some ocean parameter, but I did have a lot of fun with my daughters and girlfriend. I was really tempted to just remove NZ to mess with my mappy friends! But left it in instead โค๏ธ
Looking at
@USGSLandsat
9, and I thought I'd compare with 8. Check this out! The Landsat 9 scene is the chunk on the right, and is a square. It's overlaid on the Landsat 8 scenes. It's nearly seamless! Pretty cool ๐
I can now run a query using Dask on
@OpenDataCube
across all the pixels on Digital Earth Australia's Landsat 8 GeoMedian and calculate the average colour of all of Australia. It's light orange (I think!).
18.8 billion pixels averaged in 2 minutes over 10 servers.
We're hiring three data engineers to come work on shiny Earth observation tech in a lovely, diverse crew of scientists, software developers and remote sensing professionals. It'll be a hoot!
Pretty sure I have indexing all the 10 m global ESRI Land Cover product working in
@OpenDataCube
.
It's an awesome example of open, interoperable data! We can compare our Australian land cover product with it in notebooks next week :-)
Hey folks, us
@OpenDataCube
folks are running a conference and for the first time, we're broadcasting to you! We have one session streaming during Australian day time and one during US/Africa day time, so a timezone for everyone! Come join us ๐
You can tell from this image that the imagery here was captured in 2017 and this region is experiencing some pretty serious recession. Around 25 m a year!
Coming to the end of this coastal modelling exercise... hopefully can talk about it more soon.
If you overlay a 40 year summary of surface water on a 40 year mean (for March) NDVI, the result you get is quite pretty.
The
@DEarthAfrica
platform is really hitting its stride. Over 3 PB of data is available, including some fantastic deep time analyses.
Playing around with different land cover products over Africa. We have
@DEarthAfrica
's Cropland probability,
@ImpactObserv
's and then
@ESA_EO
's. (Sorry I don't have the categorical legends working in Jupyter!) This is over Cairo.
I'm presenting later today as part of the Cloud Native Geospatial Outreach day. I'm hosting
@sarasomewhere
's session first, at 5 pm, and then talking about
@DEarthAfrica
at 5.30 pm, AEST, including far too many live demos! Find out details here:
There's a position open on my team, where you get to have fun wrangling petabytes of open data, whilst working on open source software and having a hilarious time with the lovely folks in
#DigitalEarthAU
Check it out:
@mdsumner
@DuncanBSS
You know the weirdest useful shortcut on Mac OS? <command> + <ctrl> + <shift> + v is "paste without formatting". I have know idea how I found that, but it's super useful!
Learning how to use sat-search with
@developmentseed
and it's pretty cool. This is a calendar view of satellite data available in public APIs on the command line!
#FOSS4G2018
I was interviewed for the
@MapScaping
podcast. We talked about cloud native geospatial, but from first principles (or thereabouts).
I'd love to hear from less technical folks about whether this conversation helps demystify what cloud native means!
So, once again,
@postgis
blows my mind. Something that takes a long time in GeoPandas takes seconds in PostGIS. Joining 1.5 million points to 50,000 polygons? Seconds. Neighbor analysis? Seconds... Ridiculous!
@GergelyOrosz
Sounds pretty authoritarian...
Rather than learning from mistakes, just fire the one person who knows the most about what happened and why.
The
@USGSLandsat
's new Landsat 9 data, which was released yesterday, is now available over all of Africa in
@DEarthAfrica
.
Data from just a few days ago is available, for example, this scene over South Africa from the 8th of Feb 2022.
Did you know that
@DEarthAfrica
has 2.9 PB of freely available Earth observation data covering all of Africa?
You can discover it here:
And we have a STAC API here:
Or an interactive map here:
The
@OpenDataCube
is now an
@OSGeo
Community Project ๐
It's been great to work on this project the last few years and recognition of the project by OSGeo is a fantastic milestone for us. Onwards to full Project status!
There's a new release of the
@OpenDataCube
! Meet version 1.8! It does a heap of nice stuff. We're also on the road to becoming an
@OSGeo
project. Check it out here:
@0xhellz
Thereโs Sentinel-2 data being captured constantly. With the two satellites, Sentinel-2A and B, every part of the Earthโs land surface is captured every 5 days with a resolution of 10 m.
So much happening next week for
#geoweek19
. We've been doing a bunch of work preparing data and visualisations for
@DEarthAfrica
, so make sure if you come and hear about it on Tuesday! And you can view some of our work right now in the DE Africa Map:
I spoke to a family member today and their family business, a cafe of 20+ years, has had to let go 20 loved staff members because there is simply no way to pay them any more.
Just wanted to share, because being able to work from home with kids is an extraordinary luxury.
I received my last pay check from the University of Tasmania today, and itโs pretty humbling, as I donโt have a next thing lined up.
But Iโm grateful for my experience working there and excited about what lies in the future.
Ok, twenty years of daily rainfall data over Africa animated using the
@OpenDataCube
on
@DEarthAfrica
! You can explore the data too using our Sandbox, or discover it on our STAC API ๐
Itโs my last day at
@_FrontierSI_
today and Iโm really looking forward to doing even more with the
@OpenDataCube
at
@GeoscienceAus
starting from next week! Hoping the gif is right ๐
Hey folks, find out what we do in
#digitalearthaustralia
at our public showcase! Complaints about data access? Come. Want to know what data is available? Come. Want to see someone like
@SatelliteSci
or
@Claire_Science
talk about amazing science? Also come!
This one has a bit more detail in it!
Just pushed a 88 GB, 2 m resolution DEM of Tasmania up to
@source_coop
so it's a bit more easy to access than it used to be. Direct access is available via
An 8 bit model of the flooding in Sydney yesterday. Magically produced by one of our data pipelines from near-real time Sentinel-2 data. Thanks to
@IndecisiveMatt
for being decisive and making this work!
Worked out how to stream the
@NASA
and
@USGSLandsat
HLS data straight out of Earthdata and it works really well!
Below is a three-month cloud free mosaic of Precipitous Bluff in Tasmania.
Notebook is here:
The
@OpenDataCube
's Cube in a Box has now got automatic indexing of the
@esriaustralia
Land Cover dataset, so you can work with global Sentinel-2 and Land Cover data in minutes on your local machine or an EC2 instance:
Our
@FOSS4G_Oceania
conference registration desk is open at the Auckland University of Technology!
So much happening this week, and a great vibe among workshop participants today!
#foss4g23
Drunken wisdom from Reddit. Priceless!
"Good code is code that can be understood by a junior engineer. Great code can be understood by a first year CS freshman. The best code is no code at all."
Just consuming 2.7 TB of memory and 1,500+ CPUs crunching data. There's a point at which AWS, Kubernetes and all the other little pieces of learning curve become worth it! Should be done by Monday ๐
In case you're not aware, we're running the Free and Open Source Software for Geospatial and State of the Map conference later this year. And you can now get involved by sponsoring the event. Check out our sponsorship prospectus here:
@FOSS4G_Oceania
Fucking fantastic! This is now the primary reason I regret not being able to be at the event.
You know when the organisers can assemble a panel like this, theyโre real forward thinkers and are building towards the future we want.
Hey folks, I've put some documentation down about how you can access
#digitalearthau
data directly from AWS' S3. Also included are notes on SNS notifications for new data.
I'd love feedback from anyone this is useful for, or what's missing!
#eochat
A clear example of the benefits of analysis ready data. Left is a water index run on the
@awscloud
Landsat PDS level 1 data, and on the right the same index run on
@USGSLandsat
level 2 data. Thanks to Caitlin on our team at
@_FrontierSI_
for creating this using the
@opendatacube
.
We have a role open for a Geospatial Data Engineer. Come work with our team of fancy folks and help us wrangle data for Australia and Africa! Check it out here and please ask me questions:
Workshop spotlight: Do you want to understand how to access and enable analysis of over 30 years of earth observation data? Get in quick and join
@alexgleith
for his workshop on using Digital Earth Australiaโs Open Data Cube:
Around 18 months ago we embarked on rebuilding an urban planning application using
@Mapbox
and we weren't sure it would work, but it did, and it's super pretty! (Built with
@vuejs
,
@turfjs
, MapBoxGL etc.)
I'm excited to be working with a great team both in
@spc_cps
and
@D4DInsights
to deliver something meaningful for people in the Pacific.
It's an exciting program with new challenges and some looming deadlines, all of which will be fun to tackle!
I'm thrilled to welcome
@alexgleith
to
@D4DInsights
to support the establishment and launch of the
@DEarthPacific
program at
@spc_cps
. His previous experience with Digital Earth will be a huge asset for the Pacific.
Great presentation from
@Steven_Ramage
on the work of GEO taking Earth observation from data to decisions. Itโs really good to hear about their support for the
@OpenDataCube
and
@DEarthAfrica
along with
@GeoscienceAus
. I have a talk on the ODC on Friday if you want to know more!
Nikita from
@DEarthAfrica
has produced a cloud free mosaic over all of Landsat 5 and 7 over Africa. This image is from 1984, over Cairo viewing three measures of variance. This was made possible by a lot of hard work from
@Kirill_kzb
as well as a bunch of hardcore science folks!
One of the things we do at
@osgeo_oceania
is support the Oceania open geospatial community through our grants program.
This picture attached is from a Women in Mapping event in Kiribati, which we are stoked to be able to support!
Find out more here:
Hey web mapping Twitter, does someone know how to make all the tiles in a Leaflet or MapboxGL animated PNG map sync up? Currently it's just random and silly!
Map:
Code:
We've spent $6,000 USD on SPOT EC2 instances doing some work for
@DEarthAfrica
that we will talk about soon. That may not sound like a lot, but it's a huge amount of computation! Fun times ๐
You can now download version 1 of the QGIS ICSM Transformer plugin and run precise transformations between Australian datums using official transformation grids. Get in there and test it!
@qgisau
@ICSM_gov_au
#augistribe
We have a job going in
#digitalearthau
. This role leads the cloud team in building platforms that make large volumes of Earth observation data more easily accessible. More importantly, we're a diverse team of people working on
#opensource
and
#opendata
!