Organisers have called for former Test batsman Greg Ritchie to be removed from a cricket tournament featuring Muslim players in response to claims of racist and anti-Muslim comments.

Ritchie was scheduled to play for the Australian Cricket Association (ACA) Masters in a Twenty20 match against a World Defence XI team on November 24 at Canberra's Manuka Oval.

The match is the culmination of the International Defence Cricket Challenge, which starts on Wednesday and runs until November 23 and includes teams from the Royal Malaysian Air Force and the Pakistan Armed Forces.

Ritchie - who played 30 Tests between 1982 and 1987 - has been in the spotlight since comments he made during a lunchtime speech at the Brisbane Cricket Ground Trust members dining room on the first day of the Test against South Africa on Friday.

"I've got nothing against the Muslim people," Ritchie was reported as saying on Friday.

"Just this morning I had to try and stop three little Muslim boys trying to break the lock on my car boot.

"I had to say, 'Shut up! You're in there for a reason!"'

Ritchie also came under fire for accusations of racism by using the word "kaffir", which is a highly-offensive description of black people.

Tournament director Lieutenant-Colonel James Brownlie contacted the ACA on Monday and voiced his disapproval at Ritchie's presence given the large number of Muslim players involved in the Malaysian and Pakistan teams.

"We don't want Greg coming to the tournament," Colonel Brownlie told Fairfax.

"I don't know the exact number, but there's somewhere in between 35 and 40 Muslims competing in the tournament.

"The last thing I want is someone who has said something that could possibly degenerate what we're trying to achieve.

"They (the ACA) have agreed.

"I don't think they were aware there were so many Islamic people involved in the tournament, both from Malaysia and Pakistan.

"Once they understood my sensitivities, it was hardly an issue for them to make that call.

"It would have been difficult politically and culturally to have Greg come to the final, when our tournament motto is 'defence unity through cricket'."

ACA chief executive Paul Marsh said a decision on whether Ritchie would be withdrawn from the match had yet to be made.

"At this point in time we haven't had the chance to speak to Greg and we'll do so over the coming days," Marsh told Fairfax.

"From there I guess a decision will be made."

Ritchie has defended his comments, saying they were used as part of a light-hearted public speaking act.

"That's a joke that I use, and I'll continue to use it," he told Fairfax of the Muslim comment.

"It's just a little humorous joke to indicate that they're not my favourite people of my choice.

"If they take offence, that's their choice."

Cricket Australia has responded by ordering Ritchie to refrain from giving speaking engagements at Test venues.

Before making headlines, Ritchie had been heavily involved in charity work with the ACA Masters for many years since retiring from cricket.

"For all the criticism going Greg Ritchie's way, understand that he has raised many thousands of dollars for grassroots cricket over many years with the same style of speech," Marsh wrote on Twitter.

"I know Greg well, I don't believe he's racist and he's done plenty for grassroots cricket."

Cricket ACT declined to comment on whether or not Ritchie should be allowed to take part in the match because it was not an event they had organised.