Gray started his Rugby career at Nottingham, he was part of the Nottinghamshire County Schools Rugby Club and also played with Notts, Lincs & Derby County side. Gray took part in a tour of Australia with Three Counties at the age of 15. He played for England under16's A and played for the England under 18's the following year while he was still only 16.

Gray won the Churchill Cup with the England Saxons after starting the Challenge Cup final for Harlequins that they won in 2011.

He started for Harlequins in their 2011–12 Premiership final victory over Leicester Tigers. Following his 80 minutes in the final he toured with England that summer to South Africa and played in the two midweek uncapped games against the Barbarians.

On 27 January 2019, Gray faced his old club Harlequins in a Premiership Rugby Cup match, against Saracens. An away win for Saracens, in which Gray came on in the second half as hooker. He was a replacement as Saracens won the 2019 European Rugby Champions Cup Final. He returned to Harlequins on loan in January 2020 before signing full time with the club.

Gray won his second Premiership title, this time as a replacement, as Harlequins won the game 40-38 in the highest scoring Premiership final ever on 26 June 2021.

Gray got his first taste of club coaching with KCS Old Boys RFC in the 2015-16 season before going on to become the Director of Rugby at Barnes RFC.

Following his retirement from playing at the end of the 2021–22 season. He became head coach at London Scottish.

Post rugby, Joe started a company with his wife called Myo Masters, aimed at helping athletes at all levels have access to the best and latest tools to aid with their recovery. Started by accident whilst working on his kitchen remodel, Joe created their first massage gun with a power jigsaw and a table leg. Taking it to Quins and being mocked by Kyle Sinkler, only to catch Sinks using it and actually buying it off Joe - Myo Masters was born and has since gone from strength to strength